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TITI.E:  Acqui^iltion  Minagement  of  Electronic  Warfare  Systems 
Author:  Frederick  L.  Westover,  Lieutenant  Colonel,  USAF 

The  Air  Force  continues  to  be  challenged  In  acquiring  the  electronic 
wdrfare  systems  needed  to  counter  the  future  threat.  Fundamental  to  this 
challenge  is  the  need  to  advance  the  state  of  the  art  In  technology  in  order 
tn  Deat  the  threat  while  controlling  technical,  schedule,  and  cost 
performance,  lo  date,  our  tracK  record  Is  not  Impressive.  Across  the 
board,  systems  are  oeing  procured  that  fall  short  of  user  requirements,  are 
years  behind  schedule,  and  are  experiencing  gross  cost  overruns.  Can 
anything  be  done  to  improve  the  acquisition  management  of  electronic  warfare 
systems?  Vne  search  for  the  answer  has  been  difficult  but  provides  some 
basic  insights  into  the  complexity  of  the  acquisition  process  and  identifies 
the  essential  eionients  needed  to  control  program  performance.  The  answer  to 
successf u I  acquisition  management  can  be  summarized  as  follows: 

1.  An  experienced  government/contractor  team  Is  required  that  has  a 
strcciy  tnanagement,  cystems  integration,  and  technical  background. 

2.  The  team  must  make  tne  necessary  resource  commitments  to  the 
pi  agram. 

;i.  Tne  team  must  have  In-place  an  organizational  infrastructure  and 
the  discipline  to  follow  an  orderly  and  controlled  acquisition  process. 

4.  Concurrency  between  development  and  production  must  be  reduced,  arc 
a  rriinageirent  plan  Implemented  that  tics  key  program  decisions  to  the 
contractor  meeting  performance  milestones. 
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INTRODUCTION 


All  Koi  ce  Sy'jtem;  Command  continues  to  be  challenged  in  acquiring  the 
electronic  warfare  (EW)  systems  needed  to  counter  the  future  threat. 
Fundamental  to  this  challenge  is  the  need  to  advance  the  state  of  the  art  in 
technology  In  order  to  beat  the  threat  while  controlling  technical,  schedule 
and  cost  performance.  To  date,  our  track  record  Is  not  impressive.  Across 
the  board,  systeins  are  being  procured  that  fall  short  of  user  requirements, 
are  years  behind  schedule,  and  are  experiencing  gross  cost  overruns.  Over 
the  last  few  years  much  has  been  written  about  the  systems  acquisition 
process,  und  a  host  of  new  policies  and  regulations  to  control  the  process 
have  been  Implemented.  In  this  paper  I  address  the  question  "Can  anymore  oe 
doi.e  to  improve  the  acquisition  management  of  electronic  warfare  systems?" 
Based  on  my  research,  successful  acquisition  management  can  be  summarized  as 
fol lows: 

1.  An  experienced  government/contrc.ctor  team  is  required  that  has  a 
strong  management ,  systems  integration,  and  technical  background. 

2.  The  team  must  make  the  necessary  resource  commitments  to  the 
pi ogrum. 

3.  The  team  must  have  In-place  an  organizational  Infrastructure  and 
the  discipline  to  follow  an  orderly  and  controlled  acquisition  process. 

4.  Concurrency  between  development  and  production  must  be  reduced.  a:.d 
d  n.ari  igeinent  plan  .n.plfmeriteJ  that  ties  key  program  decisions  to  the 
contrncto:  meeting  performance  milestones. 
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After  settinn  the  stage  In  terms  or  the  EW  ehallenq?,  the  following 
fundamental  areas  of  the  acquisition  process  will  be  exciiiined  in  terms  rjf 
prcciems  ana  steps  reeded  for  successful  program  management: 

1.  Development  of  program  requirements. 

2.  Source  selection. 

3.  Government/contractor  team. 

4.  Program  control. 

5.  ?ull  scale  engineering  development  (FSED). 

S,  Transition  Into  production. 

In  this  paper  the  observations  and  research  are  based  on  the  author  o 
experience  on  the  F-4G  Wild  Weasel  Performance  Update  Program  (PUP),  the  KW 
Area  Reprogramming  Capability  (ARC),  the  B-l  defensive  avionics  program,  nnc 
the  EF-111  upgrade  program.  While  EW  is  specifically  addressed,  the 
problems  as  well  as  the  conclusions  and  ’"econimendat Ions  apply  to  other 
system  acquisition  programs  as  well. 

For  the  purpose  cf  this  paper,  the  lol lowing  assumptions  have  been 

made: 

1.  The  acquisition  manager's  job  will  not  become  easier  in  the  futjre. 

2.  The  EW  threat  will  be  difficult  to  quantify. 

3.  Emphasis  will  be  placed  on  high  technology  to  offset  Soviet 
numerical  advantages, 

4.  Pressures  will  exist  to  shorten  the  acqulsitioi.  v.vu!e. 

5.  Congressional  oversight  will  continue. 

6.  Defense  budgets  will  continue  to  decrease. 
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THE  ELECTRONIC  WARFARE  CHALLENGE 


'Oi'‘ 

The  Soviet  U'ljori  lemiins  the  numoer  one  threat  to  U.S.  security  and 
naiional  i  ii  ter  est  n .  ( 1  j  6)  Over  the  last  two  decades,  trie  Soviet  EW  threat 
nas  significantly  Increased  In  terms  of  threat  types  and  signal  densities. 

As  compared  to  the  thteat  of  the  late  60's,  the  number  of  different  types  of 
threats  (i.e.  early  warning,  target  acquisition,  target  tracking,  and 
missile  guidance  syriiems)  has  increased  three  fold,  and  the  signal  density 
has  increased  thiity  told. (2)  Also,  the  threat  has  advanced  to  include 
emitters  that  have  incorporated  low  probable  Intercept  (LPI)  emission 
techniques  whicn  make  detection  by  our  current  EW  systems  extremely 
difficult.  LPI  emitters  operate  with  narrow  beaip  widths,  short  transmiss.rr 
ti.mrs.  ana  sophisticated  waveforms. 

The  advanced  Soviet  threat  in  terms  of  threat  types,  signal  densities, 
and  LPI  techniques  requires  our  future  EW  systems  to  respond  much  faster 
than  current  systems  by  Incorporating  highly  complex  receiver,  processor, 
and  jamming  capabi I i t ies.  In  other  words,  we  must  push  technology  to  beat 
the  threat. 

Some  critics  of  the  EW  acquisition  process  believe  the  Air  Force 
ovei states  the  threai  which  results  in  high  risk  system  designs  that  canr.ot 
he  implemented  within  program  cost  and  schedule  constraints.  Iney  argue 
tint  direct  curifliot.  with  the  Soviet  Union  is  extremely  unlikely  and  our 
focus  should  be  tounrds  low  intensity  conflicts  In  the  Third  World  against 
"less  sophisticated  throatj".  Future  conflicts  would  be  similar  to  v’hat  the 
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Israelis  encountered  against  Syrian  air  defense  systems  In  1982  or  what  th>' 
U.S.  eynerienced  against  Libya  in  1986.  In  both  cases,  5W  systems  developed 
in  Che  late  60's  and  early  70's  were  used  to  counter  a  somewhat  dense  ri'.jt 
unsophisticated  Syrian  and  Libyan  threat  environment. 

The  outcome  of  both  conflicts  were  embarrassments  to  the  Soviets,  who 
have  since  rearmed  Syria  and  Libya  with  more  and  Improved  air  defense 
systems.  In  the  future,  we  cannot  assume  that  the  Soviets  will  only  export 
older,  less  sophisticated  systems  to  their  client  States.  Also,  both 
contiicto  occurrea  over  a  short  duration  where  the  elements  of  ouiprlse  'i 
tartics  played  an  Important  role.  Today  the  enemy  Is  better  prepared,  aio 
our  success  rate  would  decrease,  particularly  when  faced  with  the  incre.iofu 
signal  densities.  Finally,  the  EW  acquisition  process  requires  10  to  12 
years  to  field  a  new  or  improved  system.  The  SA-b  arid  S^-6  that  were  usf.i 
by  Syria  and  Libya  required  the  U.S.  to  advance  technology  in  the  late 
60 's  and  early  70' s.  For  the  future,  the  SA-lO,  SA-11  and  SA-12  require  the 
U.S.  to  advance  technology  today.  Given  the  time  it  takes  to  field  a 
system,  we  must  push  technology  now  in  order  to  defeat  the  threat  of  the 
future. 

While  it  is  unlikely  that  the  U.S.  will  fight  the  Soviet  Union,  it  r. 
possible  that  we  will  encounter  a  future  Soviet  threat  (i.e.  5A-10)  in  a 
third  world  country  that  our  current  EW  systems  cannot  defeat.  We  have 
overstated  the  threat,  and  a  reality  of  EW  Is  high  technology.  The  protuei.i 
we  face  in  acquisition  Is  managing  the  risks  associated  with  implement inu 
hi  an  technology. 
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The  tiW  technicdi  chaMenge  can  be  translated  into  some  basic  system 
capabilities  that  drive  hardware  and  software  requirements.  Typically,  new 
techfiology  is  needed  to  meet  these  requirements  in  terms  of  speed, 
packaging,  weight,  and  power  consumption.  Examples  of  new  technology 
include  the  development  of  microwave  integrated  RF  circuitry  to  obtain 
ml n 1  <jtur izat ion ,  gate  arrays  to  reduce  size  and  increase  speed,  low  power  PF 
mixer  amplifiers,  and  components  that  perform  over  broad  RF  bandwidths. 

Also,  EW  systems  are  integrated  into  high  performance  aircraft  which  place 
severe  environmental  constraints  on  system  operation.  System  integration 
probably  presents  the  greatest  technical  challenge.  EW  systems  consist  of 
numerous  line  replaceable  units  (LRUs)  that  Include  antenna  arrays, 
receivers,  transmitters,  rjigna!  processors,  disp^ys  and  power  sources.  For 
example,  the  B-1  defensive  system  has  over  100  LRUs.  It  is  extremely 
olff'cult  to  successfully  integrate  these  LRUs  within  system  performance 
requirements  and  without  causing  electromagnetic  Interference  among  the 
various  units.  Finally,  the  EW  system  must  have  high  reliability,  and  if 
failures  do  occur,  they  must  be  readily  identified,  isolated,  and  correctec. 

Taken  together,  the^'e  technli.al  challenges  translate  into  moderate  tc; 
hi  oh  risk  EW  progrirns.  Steps  must  be  taken  throughout  the  acquisition  c'-cie 
to  reduce  and  contr.n  these  risks. 
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THE  ACQUISITION  MANAGEMENT  PROBLEM 


Acquisition  managers  need  to  control  program  uncertainties  in  terms  uf 
technical,  cost,  and  schedule  performance.  This  becomes  extremely 
challenging  given  the  high  technology  associated  with  EW  systems. 


Defining  Performance  Requirements 

At  the  center  of  the  EW  acquisition  management  problem  is  the  need  lo 
advance  the  state-of-the  art  In  order  to  counter  the  future  threat. 
Off-the-shelf  hardware  often  does  not  meet  performance  requirements.  To 
counter  the  threat  requires  tight  performance  specifications  at  the  systei.i 
level  and  even  tighter  specifications  at  the  subsystem  level.  It  Is 
difficult  to  successfully  translate  system  requirements  into  subsystem.  LPi; 
and  board  or  module  requirements.  For  example,  does  a  design  engineer 
translate  system  performance  requirements  of  reaction  time  and  location 
accuracy  Into  designs  for  specific  RF  and  processing  circuits?  This  Is  not 
easily  done  without  extensive  modeling  and  breadboarding.  Even  then,  the 
performance  is  uncertain  until  actual  hardware  Is  developed,  integratect  irn; 
testeo. 


gQre_c^5ting  Cost  and  Schedules 

Due  to  the  aesign  uncertainties,  acquisition  managers  have  been  uri.ole 
to  accurately  forecast  program  cost  and  schedules.  Past  track  t ecof ds  fur 
EW  systems  show  that  they  experience  significant  cost  overruns  anti  are  v‘>at  • 
behind  schedule.  For  example,  in  1987  the  receiver  being  developed  unue: 
the  F-4G  Wild  Weasel  PUP  wac  projected  to  incur  a  cost  growth  and  .1  itin 
year  slip  in  production  del  I ver les. ( 14)  Gome  may  argue  that  managers 
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rle  I  1  t^er  jte  I  y  undi^rest  im<ite  cost  and  schedules  in  order  to  sell  their 
programs.  As  the  argument  goes.  If  the  truth  was  known,  no  program  wouia 
ever  receive  support  trom  Congress.  While  acquisition  managers  are  not 
dishonest,  they  tend  to  be  overly  optimistic  in  estimating  cost  and 
^5chedule3  and  tend  to  underestimate  the  risks  associated  with  developing  new 
technology.  For  example,  they  may  assess  the  technical  risks  as  low  to 
medium  when  in  fact  the  risks  are  medium  to  high  and  estimate  their  cost  ana 
schedules  on  the  former  assessment. 

Asses slriQ  Program  Risks 

Over  the  last  several  years,  much  has  been  written  about  the  systems 
acquisition  process,  and  a  host  of  new  policies  and  regulations  to  control 
that  process  have  oeen  implemented.  While  the  Jury  is  still  out  on  how  well 
these  reforms  are  working,  preliminary  Indicatlorjs  are  that  not  much  has 
changed  in  our  ability  to  control  the  acquisition  process. 

Even  though  manage-'s  try  to  do  everything  possible  to  shorten  the 
acquisition  cycle,  experience  shows  that  it  still  takes  10  to  12  years  to 
field  a  new  or  Improved  EW  sy3^em.  It  can  take  even  longer  if  attempts  are 
made  to  unrealistically  compress  schedules,  understate  technical  risks,  or 
to'^e  v:hortcuts  rtuiatd ng  system  development,  integration  and  test.  Unde: 
thesi-  c i f cumstancfs .  u  system  may  meet  delivery  schedules  but  not  meet 
per  f  (iL  mance  requ  i  r  eiiic:it  o.  To  correct  these  performance  deficiencies 
reuuires  extensive  modifications  at  significant  additional  cost.  Steps  Ca:. 
be  to  keep  riSKs  at  a  medium  level,  but  schedule  compression  and  snc. 

cut'j  reg.-irdlng  syutem  development,  integration  and  test  only  increase  risr- 
Hid  h.ivc  little  chance  of  success. 


Assessing  a  program  realistically  in  terms  of  technical  risks  and 
estimating  cost  and  schedules  accordingly  would  be  an  important  step  in 
control  ling  the  EW  acquisition  process.  For  planning  purposes,  technical 
risks  should  be  assessed  as  medium  to  high,  schedules  should  be  based  on  a 
10  to  12  year  acquisition  cycle,  and  budgets  should  be  In  line  with 
technical  risks/  schedules.  (Recommendation  Rl) 
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DEVELOPMENT  OF  PROGRAM  REQUIREMENTS 


0e^i2Pj[Lejil  of  Technical  Perfonnancg  Requirements 

In  order  to  develop  system  performance  requirements,  the  threat  must  be 
defined.  This  definition  Is  largely  based  on  assumptions  and  projections 
from  the  Intelligence  community.  In  some  cases,  the  Job  is  made  easier 
through  exploitation  of  actual  Soviet  equipment  or  collection  of  electronic 
Intelligence.  Unfortunately,  little  quantitative  data  exists  that 
accurately  describes  the  future  threat.  Consequently,  performance 
requirements  are  often  defined  in  terms  of  worst  case  scenarios.  Various 
threat  scenarios  can  be  modeled,  and  performance  requirements  can  be  deriveo 
from  the  results.  Often  the  technology  does  not  exist  to  support  the 
performance  requirements,  and  the  state-ot-the  art  must  be  advanced.  System 
design  becomes  a  revolutionary  versus  an  evolutionary  process  even  though 
many  EW  programs  are  advertised  as  performance  "updates"  to  existing 
systems. 

From  the  start,  EW  programs  can  get  into  trouble  by  not  recognizing 
that  significant  design  Improvements  are  needed  to  counter  the  threat. 

Given  the  threat,  EW  systems  are  being  designed  to  include  expanded 
frequency  coverage,  monopulse  processing  capabilities,  complex  waveform 
demodul at ion/modul at  Ion  techniques,  and  accurate  location  measurement 
capabilities;  all  of  which  drive  the  state-of-the  art  in  terms  of  system 
Hiding,  packaging,  and  pov/er  consumption.  Is  all  this  necessary?  Since 
employment  of  nut  EW  systems  usually  lags  the  threat,  the  answer  is  probab.v 
yes.  Howi.'ver,  obtaining  the  required  EW  capability  will  be  expensive. 
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As  a  part  of  the  requirement  definition  process,  definition  studies  arc 
Inojstry  surveys  May  have  to  be  conducted  to  identify  the  technical  risk 
areas,  if  the  technology  is  not  in  hand,  is  not  being  developed  In  a 
government  avionics  laboratory,  or  Is  not  being  developed  by  Industry,  thei. 
serious  consideration  should  be  given  to  making  trade-offs  between  technicol 
risk,  cost,  and  system  performance.  If  it  Is  critical  that  the  BW  system  i-n 
fielded  early,  then  system  requirements  must  be  relaxed  and  performance 
shortfalls  accepted  until  the  technology  becomes  available.  On  the  other 
hand.  It  may  be  more  critical  to  develop  the  technology  and  accept  the 
schedule  delays.  If  this  approach  is  followed,  it  must  be  done  In  an 
orderly  manner  In  which  definition  studies,  risk  reduction  efforts,  and  the 
development  of  engineering  models  precede  the  development  of  qualified 
(preproduction)  systems.  (R2) 

Development  of  Cost  and  Schedule  Estimates 

Once  the  system  performance  requirements  are  defined,  then  cost  and 
schedule  estimates  for  the  entire  acquisition  process  need  to  be  developed. 
In  the  beginning  of  a  program,  this  Is  extremely  difficult  since  the 
technical  risks  are  medium  to  high.  In  some  cases  the  technology  is  not  in 
hand,  A  program  manager  knows  that  he  wi i 1  experience  problems:  the 
magnitude  and  impact  of  those  problems  are  unknown.  Nevertheless,  cost  am' 
schedules  are  estimated  based  on  modeling,  past  experience  and  industry 
inputs.  Program  managers  tend  to  be  too  optimistic  in  developing  these 
estimates  even  when  faced  with  significant  technical  challenges.  They  nijy 
believe  that  an  overly  optimistic  approach  will  help  sell  their  program,  ico 
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once  ‘itarted,  the  past  trend  has  been  to  keep  funding  programs.  In  othe' 
words,  once  a  program  starts,  the  risks  of  cancellation  are  small. 

If  during  the  development  of  cost  estimates  the  required  funds  don't 
match  the  approved  funds,  restructuring  should  occur,  and  funding  shortfalls 
should  be  worked  in  the  out  years  through  the  POM  process.  Under  these 
circumstances,  a  slower  (lower  risk)  start  up  occurs  which  Is  more  in  line 
with  the  costs  for  definition  studies,  risk  reduction  efforts,  and 
development  of  engineering  models  mentioned  previously.  A  particular  phase 
of  a  program  should  not  start  unless  the  required  and  approved  funds  match 
in  the  current  and  upcoming  budget  years. 

In  today's  environment  of  cuts  In  defense  spending,  programs  that  cannot 
stay  within  their  budget  are  being  cancelled.  The  Precision  Locating  Strike 
System,  the  EW  Area  Reprogramming  Capability,  the  self  protection  system  for 
the  F-IM  aircraft,  and  the  F-4G  Wild  Weasel  Performance  Update  Program 
represent  examples  of  programs  that  have  been  recently  cancelled  due  to  cort 
pioblums.  In  order  to  assure  the  viability  and  executabi 1 1 ty  of  EW 
programs,  program  managers  must  base  theit  cost  and  schedule  estimates  on  a 
realistic  assessment  of  technical  risks.  This  must  be  addressed  up  front 
tipfore  the  acqulsit  on  cycle  oeqlns.  (R3) 

JevelQPment  of  dP  Acquisition  Strategy 

The  outcome  of  the  developing  program  requirements  should  be  a  syste- 
level  development  specification  .and  "model"  contract.  Both  will  serve  as 
the  bisis  for  executing  the  acquisition  program  and  should  be  closely 
reviewed  by  both  ijovernmef;t  and  potential  contractors  to  assure 
/  f  isonabl eness.  We  oo  not  w-ant  a  contractor  to  sign  up  to  a  program  ihat  re 
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can  t  execute  or  for  che  government  to  Include  requirements  that  add  1;'.  i!!- 
or  ..o  value  to  the  end  item.  The  contr<icting  strategy  for  the  entire 
acquisition  cycle  from  definition  studies  through  FSED  and  production  must 
be  tioroughly  planned  and  well  thought  out  to  avoid  serious  contractual 
problems. 

Recent  government  contractual  policies  designed  to  prevent  contractor 
overpricing  and  fraud  have  created  an  adversarial  relationship  between 
government  and  the  defense  industry .(12:00)  These  policies  have  reduced 
industry's  profit  margin  and  have  required  companies  to  share  more  of  the 
development  cost  and  Investment  requirements  which  have  created  an  adverse 
shift  in  the  contractor  risk-return  balance.  General  Bernard  Randolph, 
Commander  of  Air  Force  Systems  Command,  acknowledged  that  "Industry  right 
now  is  really  being  stressed.  In  my  view,  considerably" .( 10:36)  For 
example,  a  design  competition  formerly  required  about  one  dollar  of 
contractor  investment  for  each  dollar  of  government  funding.  The  Advanced 
Tactical  Fighter  competition,  according  to  Aviation  Week  and  Space 
Technology,  would  use  “four  to  five  dollars  for  each  government  doilat,  < 
poor  payback" .( 12:00 )  Congress  and  the  Defense  Department  have  overreirtpri 
to  allegations  of  overpricing  and  fraud  of  the  early  1900’ s.  Many 
corrective  measures  are  being  Imposed  that  are  viewed  by  Industry  as 
unprofitable  contract  terms.  Consequently,  companies  such  as  IBM  are  not 
expanding  in  defense  and  some  such  as  Eaton  have  sold  their  defense 
business.  Many  companies  who  want  to  stay  in  defense  are  faced  with  some 
tough  management  decisions  for  they  see  a  rise  in  risks  not  being  offset  by 
higher  returns. 
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Defense  contractors  need  to  be  made  responsible  for  their  actions. 
However,  government  policy  reforms  are  not  the  total  answer.  In  fact,  the 
nolicies  may  nave  a  negative  impact  of  driving  suppliers  out  of  defense, 
thus  raising  ultimate  procurement  costs. 

An  acguisltlon  strategy  Is  needed  that  recognizes  the  technical  risks 
rnd  is  fair  to  both  the  government  and  contractors.  Programs  should  begin 
competitively  with  a  minimum  of  two  teams  competing  for  FDED.  Competition 
would  include  definition  studies  that  define  system  architecture  and 
potentially  high-risk  elements  of  the  system.  Also,  competition  would 
.nclude  risk  reduction  efforts  that  further  refine  system  design  together 
with  breadhoarding  and  demonstration  of  key  elements  and  subsystems.  The 
scope  of  the  definition  studies  and  risk  reduction  efforts  should  be  well 

enough  defined  such  that  they  could  be  conducted  under  firm  fixed  price 

» 

conltacts.  While  the  contractor  will  Invest  some  of  his  own  funds  due  to 
the  competitive  environment,  his  overall  contribution  at  this  point  would  be 
-mall  (l.f.  less  than  5%  of  the  FSED  costs).  Due  to  cost  constraints, 
■vleiiiicd  competition  is  probably  not  affordable,  and  one  contractor  team 
ihovilu  oe  selected  to  carry-out  FSED  and  production.  "Not  to  exceed" 
iroo.rrt  1  on  options  should  be  included  in  the  FSED  contract. 

loday  the  tendency  Is  to  let  firm  fixed  price  FSED  contracts  for 
aouranis  tn<»t  have  mediuiii  to  high  technical  risks.  The  Advanceo  Tacricai 
'.ghter  program  Is  a  good  example.  Each  of  the  two  contracting  teams  has  a 
Mrm  fixed  price  million  contract  to  build  two  prototypes  as  weil  as  a 

icDij.ii-!  avionics  demonstrator  by  1990.  The  Air  Force  procurement  strategy 
iurcc ••  ihe  contractor;;  to  shoulder  far  more  of  the  technical  and  financie, 

[  isks  that,  in  previous  new  fighter  programs.  Each  team  is  expected  to 
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invest  between  $300-8400  mi  1 1  ion. (8:35)  If  a  FSED  contract  has  medium  to 
high  technical  rls''<s.  it  ia  not  fair  for  a  contractor  to  assume  the  majority 
of  tne  cost  risks.  An  extremely  difficult  v;orklng  relationship  will  be 
created  between  the  government  and  contractor  plus  the  contractor  will  most 
likely  implement  "work  arounds"  that  result  in  a  product  that  does  not  meet 
government  specifications. 

A  fairer  approach  would  be  for  the  government  to  share  the  majority  of 
he  r?£D  cost  risks  until  technical  risks  are  reduced  to  a  lower  level.  If 
fasts  exceed  an  agreed  upon  target,  the  contractor  would  begin  his 
contribution  by  first  sharing  his  profit  (fee)  then  10%  tc  20\  of  the  cost 
••verrun.  The  majority  of  cost  risks  still  remains  with  the  government, 
vi’hile  the  government's  total  liability  Is  unknown,  the  required  funds  based 
on  tne  government's  cost  estimate  should  reflect  a  25%  to  50%  reserve  to 
offset  program  risks.  Under  these  circumstances,  a  constructive  working 
relationship  will  exist  between  the  government  and  contractor.  Also,  the 
program  has  a  better  chance  of  meeting  performance  specifications  since  the 
contractor  Is  not  bearing  the  majority  of  the  cost  risk.  If  a  contractor 
will  not  commit  to  limited  cost  sharing,  this  should  be  an  indicator  that 
>00  much  technical  risk  exists,  and  the  program  belongs  back  in  research  -irui 
development  versus  f'SED. 

Cnee  the  design  becomes  reasonably  flew  and  technical  risks  are  reduced 
-.0  Cl  moderate  to  low  level,  a  firm  fixed  price  contact  should  be  considered 
ior  tne  remaining  development,  Integration,  and  test  efforts.  This  point 
would  occur  after  the  engineering  development  model  passed  a  system 
demonstration  test  conducted  in  the  contractor's  Integration  laboratory. 


i'oEU  would  cofitlnue  with  the  development  of  qualified  (preproduction) 
ivateiio.  Productloi.  would  follow  under  firm  fixed  price  contracts.  (R4) 

This  acquisition  and  contracting  strategy  Is  not  quite  in  line  with 
current  procurement  thinking.  As  mentioned,  the  tendency  is  to  let  a  firm 
ilxed  price  contract  at  the  beginning  of  FSED  which  is  too  early.  Until  the 
design  becomes  reasonably  firm,  the  government  should  assume  the  majority  of 
cost  risks,  if  we  force  the  contractor  to  accept  these  risks,  we  are  likely 
:o  get  a  product  that  falls  short  ol  technical  performance  requirements. 
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SOURCE  SELECTION 


Traditionally,  the  source  selection  process  has  been  a  long,  cumbersome 
effort  that  can  take  up  to  a  year  for  completion  and  cost  both  the 
government  and  contractors  millions  of  dollars.  Often  the  return  is  not 
vorth  the  investment.  An  obvious  outcome  of  the  process  should  be  the 
selection  of  a  contractor  team  that  has  demonstrated  the  capability  and 
capacity  to  accomplish  the  contractual  tasks.  In  many  cases  this  does  not 
happen,  and  soon  after  contract  award,  the  program  runs  into  trouble.  The 
.EW  ARC  program  experienced  serious  management  and  technical  problems  six 
months  after  contract  award.  The  contractor  team  did  not  have  the 
capability  or  capacity  to  develop  the  required  software  programs.  The 
program  was  eventually  cancelled  after  the  contractor  experienced  cost 
growtn  of  lOO^i.  , 

Current  Source  Selection .Process 

Source  selection  is  a  formal,  well  structured  process  that  evaluates  a 
contractor  in  several  criteria  areas  such  as  technical,  management, 
manufacturing,  logistics  and  cost.  A  contractor  Is  evaluated  against  each 
area  using  specific  criteria  and  standaros.  On  the  surface,  the  process 
should  work.  However,  if  a  contractor  can  put  together  an  Impressive 
proposal  plus  submit  a  low  cost,  he  stands  an  excellent  chance  of  winning 
'.he  contract. 

The  formal  source  selection  process  needs  to  be  restructured.  The 
government  should  be  able  to  select  a  contractor  within  three  to  six  months 
after  proposals  have  been  submitted.  If  the  government  has  done  its  job  in 
(j,'fir;lrg  the  performance  and  contractual  requirements,  then  three  to  six 
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!.onlhfi  IS  morn  than  adequate.  Unforturicte  I  y ,  this  usually  doej  not  happen; 
the  Government  enters  ; curce  selection  with  ill  defined  requirements  and 
contractual  !r, consistencies.  Uncjer  these  circumstances,  the  government  ana 
contractors  attempt  to  reach  understandings  through  a  formal  and  time 
consuming  process  of  contractor  Inquiries  and  deficiency  reports.  After 
several  Iterations,  the  contractors  are  given  a  final  chance  by  submitting 
their  "best  and  final  offer  (BAFO)".  By  this  time  technical  leveling  has 
(iccurred,  and  a  contractor  that  submits  the  lower  cost  stands  a  good  chance 
lit  winning.  The  EW  ARC  program  was  awarded  to  the  lowest  bidder. 

I'he  government  must  do  a  better  job  in  defining  the  critical 
ertormance  and  contractual  i  equ irements.  Prior  to  requesting  formal 
inoposals  from  industry,  the  government  package  nepds  to  be  thoroughly 
reviewed  by  an  experienced  team.  This  otten  Is  not  done.  Next,  the  pacicaop 
should  be  reviewed  informally  with  potential  contractors.  At  this  time, 

I  equ  1 '  ement  amblguiues  can  be  readily  Identified  and  resolved.  While  thi; 
process  does  take  some  time  and  resources,  it  is  done  in  a  less  constraineo, 
ess  formal  environment  which  should  not  require  as  much  time  and  resource; 
js  the  formal  contractor  Inquiry  and  deficiency  reporting  process  me.itioneo 
•  ir.'ier.  Given  a  uuaiity  i  equirements  pact  age,  contractors  should  be  able 
o  respond  by  submitting  one  and  only  one  proposal.  In  other  words,  the 
son* L actor  should  he  given  one  chance  to  respond.  The  formal  process  of 
rontf actor  inquiries,  deficiency  reports,  and  BAFOs  would  be  eliminated, 
j'tils  ;’nuld  reduce  the  source  selection  process  by  three  to  six  months.  (P5. 
b) 
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3y  reviewing  a  contractor's  proposal,  the  government  will  obtiln  soine 
insight  into  a  contractor's  ability  to  perform.  However,  this  is 
s.jperf i ci a  1  .  and  more  needs  to  be  done.  To  determine  a  contractor's  abji  .ity 
to  perform,  a  capacity/capability  review  shonid  be  conducted  at  his  plant 
curing  source  selection.  Currently,  this  may  be  done  on  a  limited  basis  but 
reeds  to  be  expanded  and  Include  a  review  of  all  critical  source  selection 
areas.  At  a  minimum,  the  technical,  management,  manufacturing,  and  cost 
areas  should  be  reviewed.  During  the  on-site  review,  a  government  team 
vouid  conduct  a  series  of  controlled  Interviews,  tour  critical  facllltie;! 
and  examine  critical  contractor  processes  and  procedures  to  determine  If  the 
contractor  can  perform.  Each  review  would  last  two  to  three  days,  would  be 
extremely  thorough  and  would  address  the  essence  of  a  contractor's  proposal. 
Actual  strengths  and  v.-eaknesses  could  be  readily  Identified.  For  a  brief 
time  the  contractor  would  be  put  under  a  microscope,  and  his  infrastructure 
fxaininen  in  terras  of  skills,  experience  levels,  procedures,  discipline, 
tcchr'cal  strengtns,  systems  engineering/'ana lysis  and  company  resources. 
Vhiie  a  contractor  can  make  a  proposal  loot,  good,  he  can  not  easily  cover  up 
\  eaKcesses  during  this  review.  (.P7) 

Also  ai;  jurt  of  inis  ce.  iew,  contrartf'i  i>  i.il  pci  forni'Uice  siiould  be 
■ar.^'ea.  in  source  ■boiection.  past  performance  often  is  not  reviewed  oi  it 
reviewed,  it's  discounted.  A  lot  of  insiaht  into  a  contract  or ' s  ability  t- 
i -f ;  or.Ti  in  the  future  ran  be  obtained  by  exomining  his  past.  The  onsite 
s"^verrment  plant  representative  and  program  offices  that  have  wurted  witn 
■  c.e  contractor  provide  excellent  sources  of  information. 

As  previously  stated.  IIW  systems  are  tremendousiy  complex  'nd  ['resent 
sigr.ificaiu  procurement  challenges.  The  government  must  obtain  insiuht  int- 
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;  I'ont.i  j(;t  or  • ability  tf;  perfurin,  which  the  present  source  selection 
(  I  of;e‘;j  often  does  not  pinvide.  We  can  be  lured  Into  false  hopes  by  a 
cuntractor  3  enthusiasm  or  by  the  contents  of  his  proposal  when  in  fact  he 
doesn't  have  tiie  Infrastructure  and  control  mechanisms  in  place  to  deliver  a 
qua.ity  product  in  the  required  quantities.  For  example,  it  is  not  uncommon 
to  find  a  technically  sound  engineering  house  that  can  build  one  or  two 
iystems  In  a  laboratory  environment  but  doesn't  have  the  capability  or 
capacity  to  transition  from  engineering  into  production.  Without  an  on-site 
i>-vle'.;.  this  may  be  very  difficult  to  uncover. 

By  doing  a  more  thorough  job  In  reviewing  the  government  requirements 
[■ackage,  by  conducting  reviews  at  contractors'  plants  and  by  examining 
(ontractoi  past  performance,  we  should  bo  able  to  realistically  assess  a 
contractor's  ability  to  perform.  If  a  contractor  does  not  have  credibility, 
fie  should  not  be  awarded  the  contract.  The  burden  rests  on  the  government 
lo  determine  this  through  the  source  selection  process. 


GOVERNMENT/CONTRACTOR  TEAM 


ihe  government’  program  office  and  contractor  organizations  can  make  or 
t-reak  a  program.  This  is  particularly  true  of  EW  programs  with  their  fast 
raced,  high-technology  activities. 

Government  Team 

Over  the  years,  the  tendency  in  the  Air  Force  has  been  to  “do  more  with 
less".  In  terms  of  EW.  we  have  Increased  the  number  of  programs  ,  but  at 
the  same  time  have  not  increased  the  number  of  government  personnel  to 
.mpiement  them.  To  make  up  for  shortfalls  in  personnel,  we  have  developed 
matrix  organizations  consisting  of  various  functional  disciplines  such  as 
(ngineering,  contracting,  financial,  manufacturing  and  logistics.  Figure  ! 
shews  the  program  organization  for  the  F-'<i  Advanced  Wild  Weasel  program 
conducted  during  1969-1978,  The  program  office  con.sistr^d  of  about  23  people 
assigned  directly  to  the  program  supported  by  a  couple  of  matrix  personnel. 
Figure  2  shows  the  program  organization  for  the  F-4G  Wild  Weasel  PUP  dur  inn 
:984-1987.  The  purpose  of  the  PUP  was  to  replace  the  Wild  Weasel  s 
iirocessor  and  receiver  subsystems  which  represent  over  75^  ot  the  Wild 
V/easei  avionics.  The  program  office  consisted  of  six  people  uho  were 
supported  by  approximately  14  matrix  personnel.  Two  trendra  should  be 
apparent;  the  relative  junior  level  of  personnel  and  the  lack  of  dedicated 
engineering  support.  In  1980,  matrix  organizations  at  Aeronautical  Gysternr; 
iJi'/islon  v/?re  directed  by  its  Commander  to  offset  limited  personnel 
I’esources.  f  1 1 

Willie  the  rrjt''::-;  n:  gan  i  zat  i  on  >JIov/s  coverage  ot  iiicv  pi  orit  <irnr;.  \!v 
ifteii  fail  to  -Jevelcp  the  expertise  and  continuity  neersstry  to  ei  t  ect  i  m' i  v 
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rijnage  ypcclflc  progtams.  Under  these  circumstances,  the  government  often 
performs  the  role  of  ciml n i slrators  and  becomes  totally  dependent  on  the 
(:ontr<i(;tor  .  The  matrix  organ  i 'zat  ions  are  probably  here  to  stay,  and 
nanagement  must  find  creative  ways  to  be  productive  within  this 
organizational  framework. 

A  primary  goal  of  management  is  to  establish  and  maintain  program 
(onlrol  of  technical,  cost,  and  schedule  performance.  To  control  a  program, 
rn  experienced  team  must  be  formed  headed  by  a  program  director.  The 
(logram  director  must  be  a  dynamic  leader  in  order  to  fight  the  budget 
Cattles  as  well  as  motivate  the  government  and  contractor  team.  From  a 
II  :.nagemf.nt  perspective,  he  should  be  thoroughly  familiar  with  the 
).  guisitlon  piocess  arid  have  a  solid  understanding  of  EW  technology. 

:V  vfver ,  hli:  leadership  abilities  are  key  to  maintaining  the  required 
iMvei nment  support  a, to  keeping  the  program  on  track.  Without  a  dynamic 
'f'aoer.  support  tor  the  program  can  quickly  erode,  and  contractor 
p( 'forrtiance  can  come  off  the  rails  within  a  matter  of  weeks.  That  is  the 
ii.ituif'  of  KW  progruins. 

Program  managers  (referred  to  as  engineering  managers  in  figures  1  and 
,u  oedlcdted  100  percent  to  a  specific  program  and  should  be  self 
r-tarte;'.  Audi  1 1  ona  1  1  y ,  for  large  EW  programs,  some  critical  positions 
exint  which  must  have  dedicated  people  either  co- located  to  the  program 
:  ttire  or  dedicated  100  percent  from  the  matrix  organizations.  These 
r-rltlcal  positions  consist  of  the  chief  engineer,  lead  subsystem  engineers, 
f.'ie  con'racting  officer,  and  the  financial  manager.  These  individuals, 
rjiong  with  the  program  director  and  managers,  form  the  core  of  the  program 
'  Hot-.  I:  these  positions  are  not  filled  with  experienced  people  who  are 


23 


cedlcated  100  percen'’  to  the  program,  the  government  will  not  have  the 
required  visibility  and  will  not  be  able  to  control  performance.  This  may 
be  one  cf  the  biggest  shortcomings  of  EW  management:  we  don't  have  strong 
enough  program  organizations  to  manage  the  acquisition  ot  highly  complex  KW 
systems.  This  situation  has  led  to  more  reliance  on  the  matrix  organization 
for  the  majority  of  support.  (R8) 

In  a  matrix  organization,  obtaining  loyalty  from  the  functionals  can  be 
difficult.  The  program  manager  does  not  control  their  performance  ratings 
and  normally  can  not  hire  or  fire.  Matrix  Individuals  typically  support 
several  programs,  and  its  hard  for  them  to  get  interested  in  any  particular 
cne .  Program  managers  must  create  a  positive  working  environment  for  matrix 
personnel  primarily  by  creating  a  team  approach,  delegating  responsibility 
whenever  possible  and  giving  credit  for  successes.  The  core  of  the  program 
office  is  too  small  and  does  not  have  enough  expertise  to  get  the  Job  dofif.- 
by  itself.  Matrix  personnel  can  and  must  contribute  to  the  program:  it's  un 
to  the  program  managers  to  provide  the  direction,  motivation  and  positive 
work  environment.  (R9) 

Over  the  years,  the  Air  Force  has  experienced  a  serious  problem  with 
keeping  engineers.  Currently,  we  have  an  adequate  number  of  Junior 
engineers,  but  not  enough  experienced  senior  engineers.  A  comparison  of  tne 
program  organizations  represented  in  figures  1  and  2  shows  that  junior 
officers  are  now  given  the  engineering  management  responsibilities  that  were 
previously  assigned  to  majors  and  lieutenant  colonels.  While  the  junior 
people  are  highly  motivated  and  dedicated,  they  simply  don't  have  the 
experience.  In  a  nigh  technology  field  such  as  EW.  this  ))as  rf.illy  degrudi'O 
< ur  ability  to  gain  visibility  into  contractor  activities  and  in  ti. rn, 
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contiol  the  technical  Derfotmance  nf  the  program.  Unfortunately,  the 
; ituatlon  is  getting  worse.  To  offset  the  Insufficient  number  of 
experienced  engineers,  consideration  should  be  given  to  hiring  civilian 
engineers  through  a  "support  services  contract".  This  can  be  expensive, 
fouever,  it  can  fill  the  void  if  experienced  government  engineers  are  not 
available.  (RIO) 

The  Air  Force  policy  of  reassigning  personnel  every  three  to  four  years 
c:ops  hurt  the  continuity  of  an  organization  and  the  government's  ability  to 
control  the  acquisition  of  EW  systems.  Within  three  to  four  years,  an 
individual  has  gained  not  only  the  required  experience  level  but  also  an  in 
depth  understanding  of  the  complex  interrelationships  within  the  EW  program. 
I'requently,  new  people  are  assigned  without  the  experience,  ana  they  make 
the  same  mistakes  of  their  predecessors.  Unless  an  individual  has  failed, 
the  Air  Force  should  keep  him  on  the  program  until  a  particular  phase  of 
tn.it  program  has  been  completed.  Reassignments  should  be  based  on 
completing  a  job  vi.tsus  time  on  station.  Steps  are  being  taken  within  AF3C 
■  1  do  t  h i s . ( R 1 1 ) 

Finally,  the  program  office  must  establish  a  close  working  relationship 
u,*h  key  government  agencies.  These  Include  higher  headquarters,  the  user, 
toe  MI.  logistics  centers  (ALCs),  and  test  agencies.  These  organizations 
i  hould  be  bi  ought  or;  board  at  the  start,  and  their  support  continued 
throughout  cnc  life  of  the  proyrim.  Representatives  from  these 
r  I  g i it  i  ons  should  support  design  reviev/s,  program  reviews,  and  technical 
r.fioroinat  ion  meetings.  Togr  ther  they  bring  critical  insights  and  concerns. 

'  h I ch  f  actec!  upon  by  the  program  office,  can  result  In  a  product  that 
rrttet  serves  the  user  both  oper at i ona I  1 y  and  1 ogl st leal  1 y . (PI 2) 
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The  problems  Innerent  In  a  matrix  organization,  the  inadequate  numbei 
ct  experienced  engineers  and  the  lack  of  ptogram  office  continuity  have 
forced  the  government  to  rely  more  on  the  contractor  to  represent  the 
oovernment's  best  interest.  Consequently  It  is  essential  that  a  positive 
'orking  relationship  and  team  effort  be  created  between  the  government  and 
contractor.  Equally  important  Is  the  contractor's  organization.  If  the 
contractor  does  not  have  the  infrastructure  to  Implement  the  program,  the 
rrogram  will  fall;  no  matter  how  good  the  working  relationship. 

As  a  result  of  the  capacity/capability  review  conrfucted  during  source 
selection,  the  adequacy  of  the  contractor's  organization  will  be  determined. 
Eecause  of  the  nature  of  EW,  the  contractor  must  have  a  strong  technical 
background  supported  by  an  experienced  systems  engineering,  analysis,  and 
integration  group.  Also  the  organization  that  has Vesponslbl 1 i ty  for  the 
n.anuf acturlng  process  Is  critical.  Manufacturing  processes  and  procedures 
must  be  in  place  along  with  the  skills  and  discipline  needed  to  follow  them. 
(R13) 

Each  contractor  has  Its  unique  organizational  structure.  However, 
critical  positions  such  as  the  program  manager ,  systems  Integration  manage;, 
cliief  engineer,  systems  integration  engineer,  and  flight  test  director 
should  exist  with  clear  lines  of  authority.  If  tiiese  critical  positions 
cont  exist,  are  manneo  on  a  part  time  basis,  or  are  buried  within  the 
contractor's  organization,  the  organization  is  flawed  and  vd  1 1  ie-ui  to 
trouble . 
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If  one  am. Hi  Li.^i-anjJXoiilC'igtor.  -Re  1  gLUfliiatui; 

The  contractual  arrangement  dictates  the  formal  working  relationship 
between  the  government  and  contractor.  Also,  the  informal  working 
relationship  becomes  equally  Important  In  getting  things  accomplished.  This 
roes  not  imply  that  contractual  direction  Is  given  through  the  informal 
working  relationship.  However,  on  a  dally  basis  discussions  of  problems, 
impacts,  and  corrective  actions  occur  so  that  both  the  program  office  and 
the  contractor  are  kept  informed.  In  other  words,  through  this  process  one 
tries  to  minimize  the  element  of  surprise  and  maintain  control.  Timely 
problem  reporting  and  corrective  actions  help  keep  the  program  on  track, 
rccas; onal 1 y ,  the  government  program  manager  should  meet  with  the  president 
c[  vice  president  of  the  company.  This  lets  the  contractor  know  that  the 
program  is  important  to  the  government.  Also  contractor  upper  level 
iiianagement  can  use  this  opportunity  to  reaffirm  the  company's  commitment. 

An  experienced  systems  integrator  with  a  strong  management  and 
technical  background  is  required  throughout  all  acquisition  phases.  For 
‘ome  EW  systems,  such  as  the  B-l's  defensive  avionics  program,  the  Air  Force 
tas  performed  the  systems  integration  role  with  disappointing  results. 
Basically,  the  government  does  not  have  a  strong  enough  technical  backgroend 
rr  the  experience  levels  required  to  perform  the  integration  role  of  large. 
uHifile,-'  EW  systems.  Typically,  this  role  rests  with  a  prime  contractor  wh-j 
■ 'jhcontt  .acts  the  various  subsystems  (l.e.  receiver,  processor,  janimer,  ann 
sispla',  ;)  (jf  the  EW  system.  Unfortunately,  few  successful  integration 
..ri  t :  a  I  tor''-  ex  :  3t  in  EW  . 
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Air  tramprs  tPnd  to  be  too  reyi:.ientec,  narrov/  focused,  and  over 
burdened  with  a  bureaucracy  that  parallels  the  government.  Often  they  an 
rot  structured  to  integrate  complex  EW  systems  into  ai-fcames.  Their 
organizations  prevent  them  from  cutting  across  the  different  functional 
cisciplines  and  integrating  the  various  program  elements  (be  it  hardware, 
software,  or  support  equipment)  Into  a  system.  Often  they  avoid  accepting 
systems  integration  responsibility  and  pass  that  responsibility  to  their 
subcontractors  through  premature  specifications.  For  example  on  the  Wild 
Weasel  PUP,  the  systems  integrator  managed  his  subcontractors  primarily 
through  detailed  subsystem  specifications.  However  In  reality,  the 
subs'jystem  specifications  could  not  be  realistically  defined  until  the 
technological  risks  were  reduced  and  the  design  stabilized.  As  a  result  the 
subcot, tractor  never  applied  a  systems  approach  to  his  design  which  resulted 
<n  a  system  that  fell  short  of  performance  requirements.  While  subsystems 
specifications  are  improtant,  the  systems  Integrator  should  have  placed  more 
emphasis  on  the  broader  systems  engineering  and  analysis  functions  before 
firming  up  detailed  specif Icatlons. (4:11-13) 

Some  EW  programs  have  assigned  Integration  responsibility  to  avionics 
houses.  For  example,  the  prime  contract  to  upgrade  the  EF-llI  jamming 
system  was  awarded  to  Eaton's  AIL  Division  versus  (I'rumman  or  Genet  il 
[ynamics,  tiie  aircraft  manufacturers.  While  this  Is  a  relatively  new 
initiative,  It  too  can  have  problems.  Avionics  contractors  typically  have 
e:'pertise  in  a  particular  field  such  as  receivers  or  transmitters.  They  a' c 
specialists  and  may  not  have  the  breadth  of  management  or  engineering 
experience  to  act  as  systems  !  ntegr-itoi  s.  The;  also  m.jv  not.  hove  the 
tes;)ii."ces  required  to  support  the  program.  Finally,  *ncy  im-iy  I.ti.i-  the 
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sr;:i,r.rl 'fy  i';!e. 

Systems  integration  contractors  have  been  successful  primarily  through 
their  c:rqanizationdl  structure  and  company  commitment  to  programs.  For  new 
-;ircrait  programs,  these  contractors  need  to  be  singled  out  for  future 
tusinesr.  (if  they  are  an  aircraft  company)  or  teamed  with  the  aircraft 
rotnpany  that  would  produce  the  new  airframe.  For  modification  programs, 
aircraft  companies  that  have  a  positive  Integration  track  record  or  systems 
integration  contractors  should  be  considered.  The  former  category  may  be  a 
( ompany  different  than  the  original  aircraft  manufacturer.  The  latter 
category  may  not  necessarily  be  airframers  or  avionics  houses  but  companies 
;.hat  specialize  in  modifying  and  Integrating  avionics  systems  into  existing 
circraft.  Systems  integration  remains  the  biggest  challenge  to  EW,  and  we 
!iust  do  a  better  job  in  selecting  a  prime  contractor  that  is  up  for  the 
lusk.  (kl4) 


PROGRAM  CONTROL 


According  to  Representative  Les  Aspin,  government  and  contractor 
n, ariaoemer. .  places  too  much  emphasis  on  meeting  schedules.  Aspin  believes 
the  key  concern  of  management  Is  to  push  the  goods  out  the  door". (7:16)  In 
atterr.pt  to  meet  schedules,  too  much  concurrency  often  exists  between 
cevelopment  and  production.  As  a  result,  we  end  up  with  systems  that  fall 
Short  of  performance  requirements  and  require  costly  retrofit  programs.  The 
E'l's  defensive  system  represents  a  good  example  of  a  program  that  had  too 
much  concurrency  and  is  now  undergoing  an  expensive  retrofit. 

Given  the  high  technology  associated  with  EW  systems,  technical 
performance  and  program  cost  need  to  be  tightly  controlled.  However, 
program  management  should  not  impose  arbitrary  deadlines  on  FSED  and 
production  schedules.  A  management  plan  needs  to  be  established  that  ties 
key  program  decisions  such  as  the  delivery  of  hardware,  the  start  of  flight 
testing  and  production  to  the  contractor  meeting  specified  performance 
milestones.  The  program  then  becomes  event  versus  schedule  driven,  which 
allows  management  to  better  control  program  risks.  The  program  should  not 
be  allowed  to  proceed  from  one  event  to  another  until  the  contractor 
satisfies  specific  performance  milestones.  (R15) 

Technical  Performance 

Often  the  government  lacks  timely  Insight  Into  the  technical 
performance  of  an  EW  system.  According  to  General  Welch,  Air  Force  Chief  of 
Staff,  the  Air  Force  was  badly  surprised  by  the  lag  In  the  development"  of 
the  B-l's  defensive  system. (3)  This  can  result  from  not  being  Involved  in 
the  development  process  and  from  only  monitoring  formal  tests.  In  order  to 
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control  technical  performance,  government  engineers  must  track  performance 
r- u  ! '/  on.  Thio  can  !)(>  accomplished  by  initiating  several  ictions  witn  the 
•  ont  [  actor  . 

The  contractor  must  implement  an  orderly  FSED  process  that  consists  of 
ta  eadboar dl ng  critical  funct ions/ subsystems  and  developing  an  engineering 
n ode  1/ prototype .  In  general,  the  latter  nas  the  form,  fit,  and  function  o: 
the  tinal  product  but  cannot  perform  under  all  environmental  conditions. 

This  process  will  reduce  development  risks  and  must  precede  the  development 
.)f  qualified  systems.  Tracking  contractor  events  associated  with  the 
ueveioprnent  and  test  of  breadboards,  engineering  models,  and  qualified 
rystem-j  should  begin  at  the  shop  replaceable  unit  (SRU)  level  and  continue 
through  the  system  level.  (R16) 

Government  engineer's  should  establish  indicators  to  track  the 
•'.niractor 's  technical  performance.  These  indicators  are  derived  from  the 
'iystem  level  development  specification  and  over  time  are  translated  into 
■.jbsystem,  LRU,  and  SRU  Indicators.  For  example,  a  system  level  indicator 
of  sensitivity  could  be  translated  to  a  noise  level  figure  at  the  SRU  level. 
Hy  doing  this,  the  government  can  track  progress  as  the  lower- level  design 
initures  into  a  complete  system.  <R17) 

Also,  the  development  of  test  equipment  and  test  ornredures  need  to  be 
1  .‘dcked  both  at  the  informal  and  formal  testing  levels.  Frequently  the 
.1  velopment  of  test  equipment  and  procedures  lag  the  development  of  the 
P  ime  i.'quipment.  Consequently,  a  contractor  will  not  have  the  required 
; -sources  l.n  place  for  testing  the  prime  equipment  which  leads  to 
.  :i  Kie()uat: ies  and  cost  ^s-ohedu  1  e  impacts.  The  contractor  should  have  his  test 
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resources  checked  out  prior  to  the  start  of  prime  equipment  testing;  tie  it 
at  the  SRU,  LRU,  or  system  levels.  (R18) 

The  government  his  implemented  a  formal  review  process  consisting  of 
preliminary  and  critical  design  reviews.  While  these  reviews  serve  a  useful 
purpose  in  determining  the  maturity  of  a  contractor's  design,  more  effort  is 
needed  in  terms  of  folIov;-up  reviews.  As  the  design  progresses  and  hardwai  e 
oecomes  available  for  testing,  numerous  design  changes  are  required  to 
correct  deficiencies  and  oversights.  This  process  needs  to  be  closely 
tracked  in  order  to  maintain  configuration  control,  quality  assurance  (UA). 
and  design  integrity.  Periodic  government/contractor  reviews  are  needed  to 
assess  test  results,  determine  the  extent  and  Impact  of  redesign,  and  vet  ify 
the  contractor  is  following  a  redesign  process  In  accordance  with  company 
practices/procedures.  For  example,  the  reviews  could  be  conducted  as 
mi n i -cr 1 1 lea  I  design  reviews  on  each  SRU  and  LRU  prior  to  releasing  them  to 
manufacturing.  The  contractor  review  team  would  consist  of  a  small  group  u! 
senior  engineers  from  each  functional  area  such  as  systems,  electrical, 
mechanical,  quality,  and  nianufactur 1 ng  and  would  be  led  by  the  chief 
engineer.  The  government  would  attend  uo  an  observer.  (R19) 

Finally,  government  engineers  need  to  be  physically  located  at  ctitirii 
contractor  fad  I  it. es.  They  can  not  gain  enough  visibility  into  contractor 
activit  as  oy  oniy  reviewing  documentation  sent  to  the  program  office  cr  by 
relying  on  governmet;t  plant  representatives  (l.e.  AFPRO)  who  serve  pi  in  ,i  iiy 
i.":  a  cot, tract  administrative  roie.  On  site  covet  age,  while  u  oomandi  no 
challenge,  will  give  the  government  the  required  insight  into  emit  tact ot 
progress,  design  stability  and  the  magnitude  of  design  problem:.;.  Undet  'hr 
marri,'.  oi  gar  i  zat  ton  ,  government  engineer-;  may  b..  reluctant  to  spend  eytendrc 
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nc'f  lods  o(  tlnif  at  a  contractor's  facility;  other  programs  they  support  wi  . 
suffer,  if  ijovernment  engineers  aren't  mvailable,  the  program  office  needs 
to  rely  on  a  "support  service  contractor"  tc  offset  this  shortfall.  (R20) 

Cost  Performance 

Formal  cost  performance  reporting  by  the  contractor  can  be  a  wasted 
effort  If  the  contractor  does  not  have  a  validated  system  In  place  or  the 
discipline  to  follow  it.  This  area  should  be  closely  reviewed  during  source 
selection,  and  a  plan  to  resolve  any  problems  Implemented.  Cost  reporting 
would  apply  to  the  prime  contractor  as  well  as  the  subcontractors  and  should 
be  conducted  under  both  cost  and  fixed  price  contracts. 

Contractor  cost  reporting  can  give  the  government  Insight  into  what  the 
contractor  believes  the  program  will  cost.  This  is  based  on  the 
contractor's  annual  update  to  his  cost  estimate  ^t  completion  (EftC)  and  by 
comparing  the  EAC  to  the  contracted  cost.  Also,  advance  warning  of  problems 
can  be  gained  by  comparing  the  contractor's  budgeted  cost  and  schedule  to 
liiu  actual  cost  and  schedule.  Contractor  tasks  should  be  broken  down  into 
■-iprcific  work  packages  with  starting/stopping  dates  and  budgets  assigned  tc 
each  work  package. 

Prior  to  the  contractor  running  into  cost  problems,  the 
go'. '^■^!lr!lan'.  ■  contraclui  ceoHi  needs  to  take  corrective  actions.  These  act.Gii 
incluoe  reducing  development  concurrency,  performing  risk  reduction  effort-, 
performing  early  env  i  ronmei,;  il  testing,  following  quality  assurance 
pract  i ces/'pi  r:cedufes  u  tJ  reduclng/el  iminut  ing  contractual  requirements  tna: 
ado  'it  tie  Value  to  tile  er.d  item.  .Ml  those  actions  are  aimea  at  uncover  ;:,u 
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urcb'ems  ^nct  tai;;nq  corrective  measures  at  earlier  ,  1  es55  costly  stacies  of 
the  design  process.  (R21) 

The  government  should  also  go  through  an  annual  exercise  to  update  the 
"most  probable  cost"  for  completing  the  progrc,;;i.  This  cost  estimate  is 
broader  than  the  contractor's  EAC  since  it  includes  all  contractor  and 
government  activities.  For  a  cost  type  of  contract,  the  government  should 
budget  to  the  "most  probable  cost"  since  this  represents  an  estimate  of  the 
government's  potential  liability  and  should  reflect  program  risks.  (RP2) 

As  with  most  high  technology  programs,  cost  growth  can  be  expected. 
Under  these  circumstances,  the  program  office  must  first  examine  zero  cosi 
growch  alternatives.  Usually  no  viable  alternative  exists  within  FSED  such 
as  reducing  the  number  of  qualified  units  or  reducing  the  amount  of  testing. 
AI!  end  items  and  tasks  within  FSED  are  required  in  order  to  maintain 
program  integrity.  That  leaves  the  production  program.  Typically,  cosi 
growth  in  FSED  is  offset  by  transferring  some  funds  earmarked  for  product ioii 
which  unaer  zero  cost  growth  results  in  reduced  production  quantities. 

Another  approach  to  offset  cost  growth  is  to  restructure  the  FSED 
program  such  that  the  required  and  approved  funds  match  in  the  current  ami 
upcoming  budget  years  The  restructure  stretches  schedules  by  moving 
certain  tasks  into  the  out-years.  Cost  growth  then  occurs  in  the  out  yi-ai  •: 
where  it  can  be  addressed  through  the  normal  POM  process.  While  it  is 
highly  desirable  to  avoid  cost  growth,  this  may  be  impossible  and  should  tM- 
worked  in  an  orderly  manner  by  first  examining  zero  cost  growth  a  1  ter n  i*  i vi’S 
and  then  by  working  any  remaining  cost  protilems  in  the  out  yeirs.  (Riiu: 
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1 shoulri  reflect  the  time  required  to  rea ;  i f>t i ca  1  1  y  implemenr 
i-hp  hiqh  terhnoloqy  a^aoci ated  wi  th  EW  systems.  If  government  emphasis  is 
placen  on  meeting  technical  performance  requirements  and  budget  constraints, 
theti  schedules  must  minimize  concurrency  between  developing  an  engineering 
model/prototype  and  qualified  (preproduction)  units  as  well  as  between  FSED 
and  production.  Schedules  should  not  be  success  oriented  but  should  have 
built  In  contingencies  that  reflect  additional  time  that  will  be  required 
for  redesign  and  retest.  Special  attention  should  be  given  to  system 
integration  and  ground  testing.  These  activities  are  the  most  complex,  but 
Wv'  consistently  underestimate  the  time  required  to  complete  them.  If  slips 
occur  during  FSED,  a  tendency  exists  to  compress  the  time  allocated  for 
integration  and  ground  testing  In  an  attempt  to  c^tch  up.  This  approach 
rarely  works  and  usually  results  in  railure  to  identify  and  correct  serious 
performance  problems.  During  flight  testing  and  production,  these  problems 
f-veiiLuilly  will  be  discovered  and  will  have  to  be  fixed  through  an  expensive 
letrolit  program.  Better  to  spend  the  time  early  on  to  properly  integrate 
and  test  the  system.  (R24) 

If  more  emphasis  Is  placed  on  technical  performance  and  cost,  that  doeo 
not  imply  that  program  schedules  should  float,  A  moderate  amount  of 
•.chedule  pressure  needs  to  be  applied  to  contractors  for  them  to  perform 
r-tf iciently .  In  some  cases,  a  lot  of  pressure  will  be  applied  in  an  attempt 
r.)  r  ef  a  critical  program  milestone.  However,  in  order  to  avoid  contractor 
cur  I.  out  cM.d  maintain  efficiency,  extreme  schedule  pressure  should  be  usea 
'uily  '.'ider  exceptional  c i rcumst.ances.  As  a  compromise  between  'trying  tc 
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get  something  out  the  door"  and  "engineering  the  de-jign  to  death",  piograrn 
schedules  should  reflect  a  moderate  amount  of  risk  (i.e.  somewtier  e  hetwc(.-n  ■ 
50%  and  75%  chance  of  success).  A  program  with  modei ate  risk  schedules  can 
be  kept  on  track  given  close  management  attention  and  contractor  commitment. 
As  problems  surface,  management  attention  should  be  applied  in  terms  of 
im.plemen ting  work  arounds,  adjusting  resources,  or  forming  special  teams  to 
soive  problems. 

A  common  problem  with  many  EW  programs  is  that  flight  testing  and 
production  start  too  soon.  We  end  up  fielding  EW  systems  that  operat i ona i i , 
ca.T  t  perform,  and  logistical ly  can't  be  supported.  In  part,  this  results 
from  too  much  emphasis  placed  on  meeting  schedules  and  pushing  too  much 
concurrency  between  development  and  production.  To  correct  this  problem, 
performance  milestones  within  FSED  should  be  completed  prior  to  the  start  of 
flight  test  and  the  award  of  production.  These  would  be  coordinated  and 
agreed  upon  by  the  program  office,  government  test  agencies,  and  the 
contractor  and  would  be  reflected  in  the  acquisition  strategy  for  the 
program. 

For  e.xample,  prior  to  the  start  of  flight  testing,  the  EW  hardware  ami 
software  should  be  integrated  at  a  ground  test  facility  (e.g.  integration 
laboratory).  Tnis  allows  the  contractor  to  test  the  system  under  a 
controlled  environment.  Next,  the  system  should  be  integrated  into  a  lest, 
aircraft,  and  checks  made  on  the  ground  to  verify  performance.  While  this 
environment  is  less  controlled  than  in  a  laboratory,  it  does  provide  more 
control  than  in  a  flight  test  environment.  Also,  all  flight  test  support 
activities  and  equipment  should  be  In  place  and  operational.  This  includes 
flight  test  instrumentation,  data  reduction  systems,  special  test  equipment. 
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maintenance  equipment  and  documentation.  Prior  to  the  start  of  flight  test, 
dll  problems  that  have  been  previously  Ident 1 f led  wl 1 1  not  be  solved  but 
should  at  least  have  corrective  actions  identified.  By  requiring  the 
contractor  to  pass  performance  milestones  prior  to  the  start  of  flight  test, 
an  orderly  transition  into  flight  test  will  occur.  <R25> 

A  similar  approach  can  be  used  to  enter  production.  Typically, 
production  consists  of  phases;  the  first  being  a  low-rate  initial  production 
(LRIP)  startup  followed  by  full  production  options.  An  LRIP  decision 
usually  supports  an  Initial  operational  capability  and  allows  the  contractor 
to  net  the  production  tooling  In  place  and  transition  into  a  production 
environment.  LRIP  could  be  thought  of  as  a  risk  reduction  effort  taken 
prior  to  full  production  award. 

For  LRIP,  some  FSED  milestones  should  be  completed  prior  to  contract 
award.  These  include  system  integration  testing,* some  environmental 
testing,  laboratory  reliability  testing,  and  some  flight  testing.  Enough 
testing  should  be  completed  to  give  the  government  development  and 
operational  test  agencies  high  confidence  that  the  EW  system  can  meet 
in  flight  performance  requirements.  For  full  production,  FSED  should  be 
uoirpleted  prior  to  contract  award.  During  the  remainder  of  FSED,  most  of 
which  would  consist  of  flight  testing,  emphasis  would  be  placed  on  fie  la 
•eliaoilily,  maintainabi 1 1 ty  and  support  equipment  associated  with  the  EW 
•ys’tcm.  By  minimizing  the  concurrency  between  FSED  and  production,  this 
appc each  should  result  in  production  systems  that  meet  the  operational  and 
'ogisilcs  requirements.  (R26) 
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To  tie  program  control  of  technical,  cost,  and  schedule  performance 
together,  the  government/contractor  team  should  implement  a  problem  track itio 
and  corrective  action  system.  This  system  cuts  across  all  government  and 
contractor  functions,  is  consolidated  between  the  prime  contractor  and 
subcontractors,  and  should  be  Implemented  at  the  start  of  FSED.  For  each 
problem  a  brief  description,  status.  Impact  and  get  well  date  would  be 
included  as  part  of  corrective  action  summaries.  The  prime  contractor  would 
maintain  the  system.  Problem  status  would  be  reviewed  at  the  var  ious 
technical  coordination  meetings  and  test  planning  working  groups. 
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FULL  SCALE  ENGINEERING  DEVELOPMENT 


The  government  knows  from  past  experience  that  FSED  consists  of 
significant  technical  risks.  Time  must  be  allocated,  and  steps  must  be 
taken  to  reduce  these  risks.  (R27) 

Risk  Reduction  Steps 

The  contractor  must  perform  the  necessary  systems  engineering  to 
translate  system  requirements  into  detailed  design  requirements.  For  those 
risk  areas,  breadboard  development  and  testing  should  be  conducted  at  both 
the  LRU  and  functional  levels.  Testing  should  be  conducted  using  informal 
procedures,  and  results  should  be  recorded  in  at  least  an  engineering 
notebook.  While  the  test  environment  remains  Informal,  some  discipline  is 
required  to  maintain  an  orderly  development  process  and  configuration 
control:  following  test  procedures  and  recording  results  help  provlae  that 
discipline.  Environmental  surveys  Cl. e.  1 1ml ted  environment  testing)  should 
be  conducted  on  the  more  complex  components  and  circuits.  This  helps 
establish  design  Integrity  early  in  FSED.  (R27a,  R27b) 

In  today's  high  technology  environment,  gate  array  and  very  high  speed 
Integrated  circuits  are  used  to  achieve  miniaturization  and  Increased 
tnrough-put .  Due  to  design  complexity,  design  implementation  and  reworK  an 
be  very  time  consuming.  Before  releasing  these  circuits  from  design  to  the 
build  process,  the  contractor  should  perform  extensive  simulation  of  botr. 
the  inter  and  Intra  circuit  operations.  Simulation  will  uncover  logic 
errors,  th.iing  errors,  and  reduce  the  number  of  rework  cycles  all  of  whicn 
i;ari  save  months  in  the  schedule.  (R27c) 
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Next,  an  engineering  development  model  should  be  built.  The  model 
i.ncorporates  the  breadboard  design  and  should  be  used  to  demonstrate  tlie 
functional  performance  of  the  end  item.  Test  emphasis  focuses  on  the  SRU, 
i-RU,  and  system  levels.  For  each  level,  test  procedures  are  developed,  test 
results  are  recorded,  and  design  requirements  are  further  defined.  A  subset 
of  the  test  procedures  will  be  used  by  the  prime  contractor  and  governfp‘’nt 
to  support  the  formal  acceptance  test  conducted  on  qualified  systems. 
Developing  test  procedures  in  conjunction  with  the  engineering  model  will 
save  valuable  time  during  formal  testing.  (R27d,  R27e) 

In  parallel  to  the  development  of  an  engineering  model,  test  equipnu'iit 
hardware  and  software  must  be  developed  in  time  to  support  testing  at  the 
SRU,  LRU,  and  system  level.  (R27f) 

Cnee  the  engineering  model  has  been  integrated  and  tested  at  the  s/sten 
level,  a  redesign  cycle  may  be  required  prior  to*bullding  qualified  systems. 
The  number  .r  magnitude  of  design  changes  may  be  so  significant  that  several 
SRUs  and  LRUs  require  extensive  redesign.  To  proceed  further  Into  FSFD 
would  incur  too  much  technical  risk.  At  a  minimum,  the  functional 
performance  of  the  engineering  model  should  be  domonstrated  before  bulldirni 
qu :il  i f  led  systems. 

When  ouiidiny  qualified  systems,  the  contractor  should  follow  sound 
manufacturing  procedures  and  practices.  Come  contractors  will  build  fTiFD 
units  in  an  engineering  laboratory  environment  which  does  not  provide  the 
manufacturing  control  to  track  design  changes  or  produce  quality  prodiicts. 

A  balance  must  be  established  between  the  engineering  and  production 
environments.  Building  a  system  in  a  laboratory  can  result  in  systems  th..' 
fail  to  perform  ^ince  quality  assurance  procedures  may  not  be  ciosely 
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followed.  On  the  other  hand,  a  formal  production  environment  can  stifle 
(iesl>n  pi  ogress  \  ‘  fully  applied  during  FSED.  Ideally,  production  qualif.ea 
pe.-sonnel  following  approved  manufactur  i  ng  procedures  should  build  the 
qualified  units.  However,  the  process  should  be  streamlined  to  allow 
red-line  drawings,  a  quick  but  controlled  approval  cycle  for  design  changes, 
and  a  reasonable  number  of  jumper  wires  per  SRU.(R27g) 

The  prime  contractor  should  provide  on-site  quality  assurance  coverage 
at  major  subcontractor  facilities.  The  QA  representative  would  be  involved 
In  subcontractor  day  to  day  activities.  Government  QA  representatives  woulo 
pat  form  spot  checks  during  the  build  process  and  Inspections  of  the  end  item 
during  formal  acceptance  testing.  This  approach  will  expedite  the  build 

by  streamlining  the  government's  formal  involvement  and  will  maintain 
control  of  subcontractor  activities  through  on-slte  coverage  by  the  prime 
contractor.  By  controlling  the  manufacturing  pr6cess  during  FSED  and 
designing  in  quality  early  on,  end  Items  will  be  better  able  to  withstand 
the  formal  environmental  and  flight  testing  that  follow.  (R27h) 

SystetTis  Integration  should  follow  a  phased  approach.  The  first  phase 
v/oiild  f.-onsist  of  integration  of  the  hardware  subsystems  such  as  the 
recf  iyer.  jammer  and  processor  sub.3y3te...s.  A  building  block  approach  wouic 
be  used  that  eventual  1/  tests  all  LRUs  integrated  on  a  test  bench.  In 
parallel,  the  operational  software  would  be  Integrated  starting  at  the 
modale  ( rrjmponent )  level  as  well  as  at  the  functional  level.  For  example, 
the  receiver  control  module  would  be  integrated  at  the  component  level,  anc 
the  capability  t(;  detect,  process  and  display  a  threat  would  be  integratec 


at  the  functional  level.  Functional ly,  softv/are  from  oe;eral  moriulcs  such 
as  executive  control,  receiver  control,  pulse  processing  and  display  woulif 
be  tested.  Software  integration  would  proceed  until  the  complete 
operational  program  was  integrated.  A  software  integration  facility  would 
be  used  that  consists  of  a  host  computer  to  simulate  the  airborne  hardware 
and  signal  environment.  Ideally  the  airborne  processors  that  host  the 
operational  software  along  with  the  EW  control  and  display  subsystem  would 
oe  a  part  of  the  software  integration  facility.  Also,  avionics  and  weapon^- 
simulators  would  be  included  to  provide  a  check  of  the  software  interf.iceo 
at  the  system  level.  (R28) 

After  the  EW  hardware  has  been  Integrated  and  has  passed  a  functional 
demonstration  test  (engineering  model )  or  an  acceptance  test  (qualified 
units),  system  integration  testing  can  begin.  This  represents  the  second 
phase  of  integration.  System  testing  continues  until  all  EW  hardware  and 
software  have  been  integrated.  An  integration  bench  is  used  consisting  of 
aircraft  wiring,  avionics  and  weapons  simulators,  along  with  the  EW 
hardware.  The  emitter  environment  can  be  generated  either  by  hardiining 
signals  aft  of  EW  antennas  or  by  using  an  anecholc  chamber  where  signals 
received  at  the  antennas. 

Once  this  phase  is  completed,  the  system  is  ready  for  Integration  inti; 
a  flight  test  aircraft.  This  represents  the  third  phase  of  integration 
testing  and  is  conducted  on  the  ground.  Testing  should  Include  continuity 
checks,  an  electromagnetic  compatibility  test,  and  limited  performance 
tests.  Consideration  should  be  given  to  using  a  facility  similar  to  the 
Navy's  anecholc  chamber  at  Patuxant  Piver,  Hd.  The  chamber  i^i  large  enouot’ 
to  house  an  aircraft  and  provides  an  excellent  signal  environment  under 
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controlled  conditions.  While  future  phinfj  exist,  the  facility  currently 
docfi  not  hnve  the  capabl ! i t /  to  test  EW  systems  that  use  phase 
i n t er f eromer er tor  direction  of  arrival  (UOA)  measurements.  However,  the 
Chamber  can  test  systei.^s  that  use  signal  amplitude  to  determine  DOA. 
Following  aircraft  integration  testing  on  the  ground,  the  system  is  ready  to 
enter  flight  testing:  the  final  phase  of  Integration. 

If  these  phases  are  not  performed  or  are  not  followed  in  a  systematic 
an(J  orderly  manner,  then  flight  testing  will  fall.  Past  experience  bears 
this  out.  In  the  future,  we  must  take  the  time  to  Integrate  the  EW  system 
Doth  in  the  laboratory  and  on  the  aircraft  prior  to  beginning  flight  test. 

LLiotit.  TgaUna 

Flight  testing  during  FSED  consists  of  development  test  and  evaluation 
(D‘i8.E)  and  initial  operational  test  and  evaluation  (lOT&E).  For  each,  a 
responsible  test  organization  (RTO)  and  test  director  are  selected.  For 
DT8.E,  the  RTO  could  be  either  the  test  wing  at  Edwards  AFB  or  Eglin  AFB. 

For  lOT&E.  the  RTO  could  be  the  Air  Force  Operational  Test  and  Evaluation 
Center  (AFQTEC).  An  efficient  way  to  conduct  flight  test  consists  of  a 
joint  Air  Force/contractor  DT8.E  and  a  combined  DT8.E/IOT8.E. 

Throughout  i'SED,  the  flight  test  team  must  be  involved  in  the  FSED 
.’recess  by  attending  meetings,  reviews  and  ground  tests  as  well  as  reviewing 
technical  documentation.  To  effectively  test  the  EW  system,  the  test  tea.:, 
must  have  a  detailed  understanding  of  the  system's  design  and  its  expected 
performance.  The  only  way  to  obtain  the  required  insight  and  level  of 
expertise  is  through  Involvement  In  the  program.  While  AFOTEC  has  done  a 
good  Job  of  this,  tire  DT&E  flight  test  community  has  not.  (R29) 
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During  DT8.E,  a  tremendous  amount  of  in-flight  data  will  be  recorded  by 
the  instrumentation  system,  most  of  which  will  never  be  reduced  or  analvzen. 
To  focus  the  trouoie  shooting  process,  a  procedure  should  be  f  >1  lowed  to 
identify  problems  that  occurred  during  a  flight  and  reduce/analyze  the  data 
associated  with  those  problems.  This  effort  should  be  accomplished  before 
eacn  mission.  Flight  test  problems  can  be  identified  through  crew 
debriefings  but  mure  importantly,  through  a  revlev;  of  the  video  recordings 
of  the  EW  displays.  From  reviewing  the  video  recordings,  problems  can  oe 
tagged  with  their  time  of  occurrence.  The  associated  data  recorded  by  thi- 
instrumentation  system  can  be  reduced  for  further  analysis  which  would  help 
guide  the  trouble  shooting  process.  Given  an  automated  instrumentation  and 
data  reduction  system,  problems  can  be  identified  and  data  can  be 
reduced/analyzed  within  24  hours. 

Often  success  in  flight  test  is  measured  ln*terms  of  flying  rate.  This 
can  represent  a  false  Indication  of  success  If  data  analysis  and  trouble 
shooting  does  not  precede  mission  planning.  At  the  beginning  of  DTRE,  a 
significant  amount  of  time  will  be  spent  trouble  shooting  problems  on  the 
ground  using  both  the  Integration  bench  and  test  aircraft.  At  test,  flyirui 
once  every  v;eek  to  ten  days  would  be  productive.  Once  the  initial  nT&K 
start  up  problems  are  solved,  then  flying  2  to  3  times  per  week  could  be 
achieved.  A  higher  flying  rate  results  in  lest  inefficiencies.  The  test 
team  can't  keep  up  with  analyzing  the  amount  of  recorded  data.  Also,  an 
insufficient  amount  of  time  will  be  available  for  trouble  shooting  and 
maintenance  activities. 

During  DT&E,  test  emphasis  would  be  first  placed  on  development  and 
then  placed  on  demonstrating  the  performance  requirements  contained  in  the 
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•jysteni  ypi.-r  i  f  i  cat  i  on .  Since  the  contractor  should  be  contractually 
responsible  for  in-flight  system  performance,  data  should  be  gathered  to 
supfujrt  the  contractor's  efforts  to  Identify  and  fix  problems.  At  the  siar: 
of  testing,  configuration  control  must  be  established  and  malntai ned: 
particularly  with  the  operational  software.  During  flight  test,  hundreds  cit 
softw.-n  e  changes  vl  i  1  be  made  to  correct  problems.  If  these  changes  are  net 
chocked  out  by  the  contractor  In  the  software  Integration  facility,  on  tne 
integration  bench,  and  on  the  aircraft  prior  to  flight  test,  then  flight 
t-.-vtifig  will  soon  get  out  of  control.  For  all  software  changes,  test 
procedures  must  be  established  and  followed,  documentation  updated,  and  the 
te-^r  team  briefed  on  expected  impacts.  The  test  director  should  have  final 
approval  on  test  configuration  and  whether  a  change  Is  ready  for  flight 
test.  Once  system  aesign  begins  to  stabilize,  then  data  can  be  gathered  to 
demonstrate  system  performance.  At  this  time  the  transition  would  begin 
into  lOT&E.  (R30,  R31 ) 

Prior  to  I0TS.E.  consideration  should  be  given  to  transferring  some  of 
thi  operational  software  maintenance  responsibilities  to  the  appropriate  air 
logistics  renter.  To  support  this  approach,  the  ALC  would  require  early 
delivery  of  the  software  support  tools  and  documentation  required  to 
maintain  the  operational  program.  The  goal  would  be  for  the  ALC  to  generate 
ar,  operational  program  to  be  used  during  I0T8.E.  Achieving  this  goal  wouic 
help  demonstrate  tne  ALC's  organic  capability  to  maintain  the  operations: 
sotiwere  for  a  p'uticular  EW  system  prior  to  the  system's  initial 
on.  ■  .:t : ono I  rapabiliiy.  In  addition,  the  ALC  could  perform  a  software 
ver  I  !  i  ca'.  1  on  and  validation  effort  foi  tfie  program  office.  (P32) 
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The  DT&E  and  I0T8.E  flight  test  communities  must  measure  performance 
using  consistent  and  agreed  upon  pass/fail  criteria.  In  the  past,  DT?.E  han 
been  conducted  using  broader  or  different  criteria  than  used  in  I0T8.E. 
resulting  in  the  EW  system  falling  lOT&E.  Prior  to  the  start  of  flight 
test,  the  pass/fail  criteria  for  DT8.E  and  I0T8.E  should  be  derived  from  the 
system  specification  and  agreed  upon  by  the  program  office  and  both  PTO's. 

If  differences  exist,  they  must  be  resolved  prior  to  the  start  of  flight 
testing.  Once  an  agreement  Is  reached,  the  pass/fail  criteria  should  be 
formally  documented  in  a  Baseline  Correlation  Matrix.  (R33) 

Similarly,  criteria  should  be  established  for  the  transition  from  DT&E 
to  lOT&E.  It  would  be  senseless  to  enter  I0T8.E  knowing  in  advance  that  the 
system  will  fall.  The  criteria  would  include  demonstrating  an  agreed  upon 
level  of  performance  during  DT&E  for  both  prime  and  support  equipment.  .1 
reassembled  operational  program  with  updated  software  documentation  on  site, 
and  validated  technical  orders  on  site.  These  criteria  would  be  document eo 
in  the  Test  and  Evaluation  Master  Plan.  (P34) 

To  support  the  LRIP  decision,  lOT&E  emphasis  should  be  placed  on 
providing  an  assessment  of  operational  performance.  Once  the  design  is 
stabilized  during  DT&E,  as  much  performance  data  as  possible  should  be  usn: 
to  support  the  lOT&E  assessment.  This  would  help  to  expedite  the  LRIP 
decision.  As  lOT&E  progresses  and  to  support  the  full  production  decision, 
more  emphasis  should  be  placed  on  evaluating  technical  orders,  support 
equipment  and  re! iabi 1 i ty/maintal nabl 11 ty .  (R35) 

A  positive  ar.d  close  working  relationship  must  exist  between  the 
program  office  and  the  lOT&E  RTO  (l.e.  AFOTEO.  Ptoduclioii  rlecirdons  are 
based  primarily  on  I0T«.E  test  results,  and  both  organizations  need  to  wor  i- 


close;/  together  to  achieve  a  successful  I0T8.E,  For  example,  as  problems 
die  idefilificd  dutinq  lOT&E,  the  program  office  must  work  with  the 
contractor  to  identify  corrective  actions.  In  some  cases,  problems  can  oe 
corrected  during  I0T8.E.  For  others,  solutions  will  have  to  be  implementec 
and  demonstrated  during  follow-on  testing  or  during  production.  The  program 
oft  Ice  must  be  committed  to  solving  lOT&E  problems,  and  the  RTO  must  be 
convinced  of  the  program  office's  commitment.  This  can  only  come  about 
through  a  close  working  relationship. 


TRANSITION  INTO  PRODUCTION 


As  mentioned,  production  v;l  1  I  occur  in  phases  that  may  include  a 
start  up  followed  by  multiyear  options  for  full  production.  To  reduce 
production  risks  and  help  assure  a  successful  transition  from  FSED,  an 
approach  that  minimizes  concurrency  between  FSED  and  production  can  be 
implemented  which  ties  production  decisions  to  the  contractor  meeting 
specified  program  milestones.  These  should  Include  FSED  test  milestones  ai 
equally  important,  should  Include  successful  completion  of  a  piuduction 
readiness  review  (PRR)  and  contract  negotiations. 

Production  Readiness  Review 

A  production  readiness  review  determines  whether  or  not  a  contracto:  : 
prepared  to  enter  production.  The  PRR  is  based  qn  the  contractor's 
manufacturing  plan  and  an  extensive  on-site  review  of  the  contractor's 
capability/capacity  to  produce  the  end  item.  The  PRR  addresses  all 
manufacturing  areas  which  Include  management,  production 
facilities/resources,  processes/procedures,  quality  assurance  and  rhii-.  Kr 
risk  areas,  the  contractor  would  be  required  to  Implement  risk  reduction 
actions  before  production  award.  (R36) 

Contract  Negotiations 

As  a  step  that  minimizes  cost  risk  to  the  government,  the  entire 
production  contract  (i.e.  LRIP  and  full  production  options)  showid  be 
ner,;otiated  prior  to  LRIP  award.  If  Initial  production  is  awat  dfs;  prior  to 
ccmpleticn  of  full  production  negotiations,  the  govrunmerit  wi  1  i  lose 
contractual  leverage  to  control  cost.  (R'JV) 
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Contractual  negotiations  are  a  long  and  tedious  process  that  can  Decor.e 
a  program  driver.  Therefore,  adequate  time  must  be  allocateo  for  the 
government  to  develop  a  requirements  package,  the  contractor  to  prepare  a 
proposal,  the  government  to  fact  find  the  proposal  and  the 
gover lunent/contractor  to  complete  negotiations.  This  process  should  begin  o 
to  12  months  In  advance  of  iinticipated  production  award. 

To  exped'te  the  formal  contractual  process,  the  government  ana 
contractor  should  jointly  review  the  requirements  package  informal  1 v  and 
reach  an  agreement  on  the  contractual  specifications,  statement  of  work, 
terms  and  conditions.  If  the  government  or  contractor  has  to  formally 
update  the  requirements  package  or  proposal,  this  will  take  at  least  six 
months.  Better  to  work  the  details  out  in  advance  and  only  have  one 
requlremenis/proposal  cycle. 

The  following  sections  address  two  sped f lc*areas,  namely  warranties 
and  preprocuremont  data,  that  have  caused  difficulties  during  contract 
negot iations. 

Wurra.iit  1  ys 

If  !  ntei  pret'-'d  and  implemented  In  its  broadest  sense,  the  Defense 
nppi  .jpr  1  at  Ions  Act  of  1VH4  makes  warranties  a  stf-ndard  feature  of  most 
fi  vd-price  prodact  ion  contracts.  (5:5-62)  Simply  put,  a  warranty  is  a 
contractor  commitment  to  deliver  a  product  that  meets  specif iec  standards 
for  I  specified  period  of  time,  in  principle,  a  warranty  makes  sense.  It 
prov.cieo  a  meaiis  for  the  government  ann  contractor  to  share  risks  with 
red  eel  to  per f or riiance  a.nri  provides  an  added  incentive  for  the  contractor  to 
design  aiin  produce  quality  product.  However,  imp  1  emerit  i  ng  warranty  can 
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be  extremely  complex  and  difficult  to  administer  if  steps  are  not  taken  to 
develop  and  coordinate  a  warranty  strategy  prior  to  production  award.  (R3fi; 

As  part  of  that  strategy,  four  areas  must  be  considered; 

1.  A  procedure  must  be  followed  for  defining  and  implementing  a 
warranty.  This  would  include  performing  studies  to  identify  candidate 
approaches,  coordinating  approaches  with  engineering,  user,  logistics,  and 
contractual  personnel,  and  obtaining  industry  comments  regarding  preliminary 
terms  and  conditions.  The  warranty  clause  would  become  a  part  of  the 
production  requirements  package.  <R38a) 

2.  Specific  performance  parameters  to  be  covered  under  the 
warranty  must  be  identified.  A  good  source  is  the  system  level  development 
specification.  Reliability  and  maintalnabi i 1 ty  are  the  parameters  typicilly 
addressed.  Others  may  include  operotional  performance  parameters  such  as 
system  sensitivity,  dynamic  range,  reaction  time*  and  system 

interfaces. (R38b) 

3.  Each  performance  parameter  must  be  accurately  measured  again  :t 
specific  criteria  in  order  for  the  warranty  to  be  effective.  Also,  the 
contractor  should  not  be  committed  to  guarantee  performance  parameters 
beyond  his  reasonable  control.  For  complex  EW  systems,  his  control  will  rn 
significantly  diminished  in  an  operational  (field)  env 1 rorment .  In  the 
field,  the  Air  Force  may  not  adequately  track  failures  and  perform  the 
analysis  necessary  to  determine  the  cause  of  the  failures.  Was  the  failuic 
iriduced  or  i/iherent?  Can  it  be  repeated  on  the  g:  ound?  Without  answer  ;■  t  ' 
tnese  questions,  contractor  wili  be  extremely  reluctant  to  conimit  to 
perform- :  -e  yuar  cinlee.-:  that  -ure  measured  in  the  !  ';e!d.  (R3Bc) 
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An  ilternallve  approach  would  be  to  measure  system  performance 
.n  a  laboratory  wnich  uoes  provide  a  controlled  environment.  For  example, 

: el  lability  can  be  measured  oy  conducting  a  test  which  consists  of  exposing 
the  oyster,  to  several  failure  fret  environmental  cycles.  Based  on  past 
experience,  the  system  would  have  to  demonstrate  a  reliability  three  to  four 
times  higher  than  the  required  field  reliability. 

4.  A  corrective  action  plan  must  be  defined  In  the  event  that  a 
performance  parameter  falls  the  warranty.  Corrective  actions  could  include 
the  contractor  providing  additional  spares  or  performing  redesign/retest  at 
no  cost  to  the  government.  If  additional  spares  are  to  be  provided,  then 
.onsidei at  Ion  should  be  given  to  sparing  at  the  SRU  versus  LRU  level.  This 
will  be  more  cost  effective.  (R38d) 

The  cost  of  a  warranty  varies  from  3  to  7  percent  of  the  production 
cor  If  act;  It  i.s  expenrilve.  To  assure  that  the  waVranty  will  be  cost 
effective,  a  well  thought  out  and  fully  coordinated  strategy  must  be 
developed  and  Implemented, 

Ikpr oneiiL  giiii 

A  general  policy  exists  for  the  government  to  acquire  the  data  that 
cri-vldes  the  capability  to  produce  the  end  Item  by  sources  other  than  the 
'u  Kjiiial  manufactui  er . (5:5-18)  When  a  sole-source  production  contract  is 
awarded,  the  govei nment  is  placed  In  the  position  of  having  to  depend  on  the 
contractor  tor  additional  units,  spares,  and  modi f icat ions .  Acquiring 
1  epr ocur  enit-nt  data  can  be  used  for  component  breakout  from  a  prime 
r.antf  ictoc  ,  to  Solicit  other  sources,  or  to  possibly  allow  the  cjovernmerc 
P- i/ii.  :e/ r.adl  fy  thr  item.  However,  a  major  disadvantage  of  rept  ocuremen; 
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deta  is  cost,  and  a  strategy  to  acquire  this  data  must  be  well  thought  out. 
(R39) 

Often,  the  prograni  office  defines  the  reprocurement  data  package 
without  interfacing  with  the  ALC  that  will  have  responsible  for  the  end 
item.  The  ALC  should  tailor  the  reprocurement  data  to  his  specific  needs. 
This  will  reduce  costs.  Secondly,  reprocurement  data  should  be  broken  out 
into  ^wo  categories;  the  first  consisting  of  detailed  engineering  and 
manufacturing  orawings  and  the  second  consisting  of  the  manufacturing 
procedures/processes  used  to  produce  the  end  item.  Drawings  can  be  dcquiird 
for  about  $100,000  per  LRU.  Acquiring  the  manufacturing 
procedures/processes  will  cost  millions  and  should  only  be  considered  as  a 
contractual  option  to  be  exercised  by  the  ALC.  Hopefully  the  ALC  will  only 
need  the  drawings.  (R39a,R39b) 

ciiiaily,  unlimited  data  rights  may  have  to  be  acquired.  Unlimited 
rights  allow  the  government  to  avoid  sole-source  dependence.  Because  of  the 
government's  Insistence  that  the  contractor  share  more  of  the  investment 
costs,  we  may  see  more  limitations  placed  on  the  government  in  using 
technical  data  delivered  as  part  of  a  contract.  Acquiring  unlimited  rights 
can  cost  millions  and  should  be  treated  similarly  to  acquiring  the 
manufacturing  procedures/processes.  It  may  be  more  cost  effective  to  remain 
in  a  sole  source  position  than  to  acquire  the  data  tights  and  m.anufactur  inw 
procedures/processes  needed  to  develop  an  alternate  source.  (R39c) 

A  complete  reprocurement  data  package  will  be  extremely  expensive. 

Most  likely  only  a  drawing  package  tailoted  by  the  appropriate  .ALC  needs  to 
bo  procured. 
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CONCLUSIONS  AND  RECOMMENDATIONS 


■'Ji''  :;.swer  tu  successful  acquisition  manageT.enT  of  EW  systems  can  oe 
summar I ned  as  foil uws : 

1.  An  experienced  government/contractor  team  is  required  that  has 
a  ;tif>nt5  management ,  systems  integration,  and  technical  oackground. 

2.  The  team  must  make  the  necessary  resource  commitments  to  the 

fx  ogr  xm. 

3.  The  team  must  have  In-place  an  Infrastructure  and  the 
discipline  to  follow  an  orderly  and  controlled  acquisition  process. 

4.  Concurtency  between  development  and  production  must  be 
reduced,  and  a  management  plan  implemented  that  ties  key  program  decisions 
Lvi  the  contractor  meeting  performance  milestones. 

Sf^ecific  conclusions  and  recommendations  are  summarized  in  the 
fol  lowing  Elections*. 


Tti e  .£ I <i ' ; Ii ca Ic  Warfare 


We  must  push  technology  in  order  to  beat  the  Soviet  EW  threat.  A 
rcdilty  of  EW  systems  is  high  technology. 

For  EW  systems,  system  integration  presents  the  greatest  technicai 
ch<i  I  1  engti . 

The  technical  challenges  translate  into  high  risk  programs. 


The .  Acqu  Laj 


Because  the  state  of  the  art  is  being  advanced,  technical 
performance  uncertainties  remain  until  actual  hardware  is  developed, 
integrated  and  tested. 


Program  cost  and  schedule  estimates  tend  to  be  too  optimistic  and  do 
not  realistically  reflect  the  technical  risks. 
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R1 .  ir’or  planning  purposes,  Icohnlcal  risks  should  be  assessed 
as  medium  to  high,  schedules  should  be  based  on  a  10  tu  ;; 
year  acquisition  cycle  and  budgets  should  be  in  line  witt' 
technical  risks/ schedules,  (p.8) 


System  perfoimance  requirements  are  often  defined  in  terms  of  wornf 
case  threat  scenarios. 

Significant  design  Improvements  are  needed  to  counter  the  threat, 
and  obtaining  the  required  EW  capability  will  be  expensive. 

R2.  As  part  of  the  requirements  definition  process,  dcfinitioi 
studies,  risk  reduction  efforts  and  development  of 
engineering  models  should  precede  the  development  of 
qualified  (preproduction)  systems.  (p.lO) 

In  today  s  environment  of  cuts  in  defense  spending,  programs  that 
cannot  stay  within  their  budget  are  being  cancelled. 

R3.  Management  should  base  their  cost  and  schedule  estimates 
on  a  realistic  assessment  of  technical  risks  which  should 
be  addressed  up  front  before  the  acquisition  cycle 
begins,  (p.ll) 

An  acquisition  strategy  Is  needed  that  recognizes  the  technical 
risks  and  is  fair  to  both  the  government  and  contractor. 

R4.  Definition  studies  and  risk  reduction  efforts  should  be 
competitive  and  conducted  under  fixed  price  contracts. 

One  contractor  team  should  be  selected  for  FSED  and 
production.  Until  the  design  becomes  firm  and  technical 
risks  are  reduced,  the  government  should  share  the 
majority  of  the  cost  risk  under  a  "cost  plus"  type  of 
contract.  When  the  design  stabilizes,  the  remainder  of 
FSED  could  be  conducted  under  a  fixed  price  contract. 
Production  should  be  conducted  under  a  fixed  price 
contract,  (pp, 13-15) 

glectlcn 

The  formal  source  selection  process  is  cumbersome,  expensive  and  may 
net  result  in  selection  of  the  best  contractor  team.  The  procerfj 
needs  to  be  restructured. 

R5.  Prior  to  requesting  formal  proposals  from  contractors,  'hf- 
government  requirements  package  should  be  reviewed  by  an 
experienced  team  and  coordinated  with  Industry.  (p.lT) 


R6.  Contractors  should  be  given  one  chance  to  respond  to  a 
proposal  request.  The  formal  process  of  contractor 
inquiries,  deficiency  reports  and  BAFCs  should  be 
(?  1  iml nated.  (p .  17) 

R7.  To  determine  a  contractor's  ability  to  perform,  a 

capaci ty/capabi 1 Ity  review  should  be  conducted  at  his 
plant  during  source  selection.  As  part  of  this  review, 
contractor  past  performance  should  be  examined.  (p.l8) 

Gpvt-rnment.'CQntrdc tor  Team 

To  control  a  program,  an  experienced  government  team  must  be  for.med 
head(>d  by  a  dynamic  program  director  and  consisting  of  program 
man^igers  wid  matrix  personnel  from  various  functional  organizations. 

P8.  In  addition  to  the  progt am  director  ard  managers,  the 
criiet  engineer,  lead  subsystem  engineers,  contracting 
(ifficer  and  financial  manager  should  be  assigned  full  time 
to  ‘he  progra.a.  (pp. 23-24) 

R7.  Progt am  managers  should  create  a  positive  working 
environment  for  matrix  personnel  by  creating  a  team 
approach,  delegating  responsibility  whenever  possible  ana 
giving  credit  for  successes,  (p.24) 

Ciitrently,  the  Air  Force  does  not  have  an  adequate  number  of 
experienced  senior  engineers  which  has  degraded  our  ability  to 
coiittol  technical  performance. 

RIO.  Consideration  should  be  given  to  hiring  engineers  through 
a.  'support  services  contract"  to  offset  the  shortfall  of 
experienced  senior  engineers,  (p.25) 

Pll.  In  order  to  maintain  continuity,  reassignment  of  critical 
positions  should  be  based  on  completing  a  job  versus  rime 
on  station.  <p.25) 

P)2.  The  program  office  should  establish  a  close  working 

relationship  with  key  government  agencies  that  can  oricg 
critical  insights  and  concerns  to  the  program,  (p.25) 

If  the  emit c actor  does  not  have  the  company  commitment  and  the 
organizational  infrastructure  to  implement  the  program,  the  program 
■;,il  fail. 

P13.  The  contractor  team  should  have  a  strong  technical 

oacHiground  supported  by  experienced  systems  engineer i. eg. 
analysis,  integration  and  manuf actur i ng  groups,  (p.26) 
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The  informal  working  relationship  between  the  government  and 
contractor  becomes  important  in  getting  things  accomp! ished  and 
keeping  each  other  informed. 

An  experienced  systems  Integrator  with  a  strong  management  anc! 
technical  oackground  is  required  throughout  all  acquisition  phases. 
Few  successful  integration  contractors  exist  in  EW. 

RH.  For  new  aircraft  programs,  successful  systems  integra'.  inn 
contractors  need  to  be  singled  out  for  future  business  or 
teamed  with  the  aircraft  company  that  woulo  produce  the 
new  airframe.  For  modification  programs,  systems 
integration  contractors,  who  may  be  different  from  the 
original  aircraft  manufacturer,  should  be 
considered,  (p.29) 

Prjc::a.m  Control 

Too  much  management  emphasis  is  placed  on  meeting  schedules. 

R!5.  A  management  plan  should  be  established  thut  tie«j  key 
program  decisions  to  the  contractor  meeting  s[>ecifitii 
performance  milestones.  (p.30) 

R16.  The  contractor  should  Implement  an  orderly  KJEd  p[  ;x:e -s 
that  consists  of  be oadboarding  ctitical 
functions/subsystems  and  developing  a;-  engineering 
model.kprototype .  (p.31> 

RI7.  Government  engineers  should  establish  indicators  to  trick 
the  contractor's  technical  performance.  (p.3n 

R18.  The  contractor  should  have  his  test  resources  in  plar^- 
and  checked  out  prior  to  the  start  of  prime  equipment 
testing,  (pp. 31-32) 

Formal  design  reviews  serve  a  useful  purpose;  however,  rime  nt-ed'  ‘r 
be  done  ir  terms  of  follow-up  reviews. 

R19.  Tf;  support  the  reoesinn  process,  nk'ni  riesign  reviev-s  i.i '  fi 
to  be  conducted  on  each  3RU  and  LRU  prior  to  relean.ito 
rhem  to  manufacturing,  (p.32) 

R20 .  To  gain  visibility  into  a  contractor's  activties, 

government  engineers  need  to  be  phyrrically  Incited  it 
criticai  contractor  facilities,  (pp. 32-33) 

Contractor  cost  reporting  can  give  the  government  insight  into  whit 
rh.e  contricror  believes  the  program  will  cost. 
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R21  .  Trior  to  the  contractor  running  Into  cost  problems,  the 
•Kjvernment/contrdctor  team  needs  to  take  corrective 
actions,  (pp. 33-34) 

P./.2.  For  .1  cost  type  of  contract,  the  government  should  buaget 
to  the  "most  ptobable  cost"  estimate,  (p.34) 

R23.  Cost  growth  should  be  worked  in  an  orderly  manner  by 

first  examining  zero  cost  growth  alternatives  and  then  by 
working  any  remaining  cost  problems  in  the 
out-years,  (p.34) 

P24.  Schedules  should  reflect  the  time  required  to  implement 
the  high  technology  associated  with  EW  systems. 

Schedules  should  minimize  concurrency  between  FSED  ana 
production,  (p.35) 

A  crimmon  problem  with  many  EW  programs  is  that  flight  testing  ana 
pioduction  start  too  soon. 

R25.  Prior  to  the  start  of  flight  testing,  the  EW  system 

tihould  be  integrated  at  a  ground  test  facility  and  into  a 
test  aircraft.  All  flight  test  support  activities  and 
equipment  should  be  in  place  and  operational,  (pp. 36-37) 

R26.  Prior  to  initial  production,  some  FSED  milestones  should 
be  completed  that  Include  system  integration  testing, 
some  environmental  testing,  reliability  testing  and  some 
flioht  testing.  Prior  to  full  production,  FSED  should  oe 
completed,  (p.37) 

1  <:■  F:ia'.  liCiL  1  (i'l  - 

The  qcve:  :.ment  knows  form  past  experience  that  FSED  consists  of 
■li.rii  r  iCtinl  lechnlccil  risks. 

P27.  Time  should  be  allocated  and  steps  should  bo  taken  to 

reduce  these  risks.  Risk  reduction  steps  should  inc  .ce 
the  fol lowing:  (p.39) 

a.  Breadboard  development  and  testing,  (p.39) 

b.  Early  environmental  testing,  (p.39) 

c.  Simulation  of  gate  array  and  very  high  speed 
integrated  circuits,  (p.39) 

d.  Development  of  an  engi neer i ng  model  and 
demonstration  of  its  functional 
performance.  (p.40) 


e.  Early  develop/nent.  of  test  procedure'.  (p.-iO) 

f.  Early  development  of  test  equipment,  (p.lt) 

g.  Following  sound  manufacturing  procedures  and 
practices  when  building  qualified  FIjKD 
systems,  (pp.  40-41) 

h.  On-site  OA  coverage  at  major  subcontractor 
facilities,  c  p . 41 ) 

R28.  Systems  integration  shoula  follow  a  phased  approach  that 
consists  of  hardware  integration,  software  integration, 
systems  integration  and  aircraft  integration.  (pp.4I  42; 

An  efficient  way  to  conduct  flight  tests  consists  of  a  Joint  Air 
Force/ccntractor  DT8,E  and  a  combined  DT8.E.''inT8.E. 

R29.  To  effectively  test  the  EW  system,  the  fliaht  test  team 
must  be  involved  in  all  aspects  of  FSED,  (p.43) 

R30 .  During  DT8.E,  test  emphasis  should  be  first  placed  on 
aevelopment  then  on  performance.  (pp.44-4b) 

During  flight  test,  hundreds  of  software  changes  will  be  made  to 
correct  problems.  . 

R31.  For  all  software  changes,  test  procedures  must  be 

established  and  followed,  documentation  updated  .and  the 
test  team  briefed  on  expected  impacts,  (p.45) 

R32.  As  a  step  towards  demonstrating  an  organic  sol  tout 

.:apability.  some  ot  the  opera' ional  f'OftWd’'e  na  i  nt  en  •ic.' 
re  apcnsihi  I  it  IPS  '•.l.culd  be  <iiven  to  the  .ypnc  tift  i  it  r  '.L(' 
prior  to  lOTAE.  (p.-tG) 

R33.  A  Baseline  Co”- re!  at  ior'  Matrix  should  be  generated  th.it 
documents  the  DT&B  and  I0T<1.E  test  critciiu.  ;p.4t>) 

R34 .  The  Test  and  Evaiuut'mn  Master  Plan  sluuilo  ir.rrlude  the 
DT’E  criteria  that  must  be  satisfied  before  starting 
I0T>1E.  :p.46) 

R35.  To  support,  initial  production,  early  I0T8.E  emF'hasis 

should  be  placed  on  assessing  operational  pt'r f nr mance . 


Ir  an.iill.'jlii-  i/t'ift.  Pifidu-tilon 

P36.  To  determine  if  a  contractor  is  prepared  to  enter 

riroduction,  a  production  readiness  review  should  oe 
conducted.  <p.48) 

.P37.  To  minimize  government  cost  risk,  the  entire  production 
contract  <i.e.  LRIP  and  full  production)  should  be 
negotiated  prior  to  Initial  production  award,  (p.48) 

R38.  Prior  to  contract  award,  steps  should  be  taken  to  develop 
and  Implement  a  warranty  strategy  that  includes  the 
following,  (pp. 48-50) 

a.  A  procedure  for  defining  and  Implementing  the 
warranty.  (p.50> 

b.  Specific  performance  parameters  to  be  covereo 
under  the  warranty.  (p.50) 

c.  Methods  to  verify  performance  against  spec. flea 
criteria.  (p.50) 

d.  Corrective  action  plans  in  the  event  system 
performance  falls  the  warranty.  (p.51) 

0 

R39.  Prior  to  contract  award,  a  strategy  to  acquire 

reprocurement  data  should  be  developed  that  includes  the 
fol lowing:  (pp. 51-52) 

a.  Strong  coordination  with  the  ALCs  that  will 
have  responsible  for  the  end  items,  (p.52) 

b.  Procurement  of  engineering  drawings  and  as 
options,  procurement  of  manufacturing 
procedures/'processes.  (p.52) 

c.  P'-ocurement  of  unlimited  data  rights  as  options 
to  the  production  contract,  (p.52) 
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GLOSoARV 


AFOTEC  A;r  Force  Operat  i on<i  1  ■rcL;l  and  EvTluation  Center 

AFPPO  Air  Force  Plant  Representative  Office 

AFSC  Ait  Force  Systems  Command 

ALC  Air  Logistics  Center 

ARC  Area  Reprogramming  Capability 

BAF3  Best  and  Final  Offer 

jOA  Direction  of  Arrival 

Dr&F  D';‘ve ! opment  Test  and  Evaluation 

LAC  F'.'ir.ate  at  Completion 

EW  Electronic  Warfare 

F3ED  Full  Scale  Engineering  Development 

I0T3.E  Initial  Operational  Test  and  Evaluation 

I, PI  Lov;  Probable  Intercept 

LRIP  Low  Rate  Initial  Production 

LRU  Line  Replaceable  Unit 

PDM  Program  Objective  Memorandum 

PRR  Pioduction  Peadir:e‘;::;  Review 

PUP  P '?r  1  ornance  Update  Pt  oui  im 

GA  Cjality  Assurance 

RF  Pad.o  Frequency 

RTO  Pesfonsible  Test  Oi  gvti  i  r.  i ‘  i  on 

SRU  Siicu  Replaceable  Cn  i ! 


